Typed datataset from Avro generated classes?

[Joaquin Tarraga]

Hi all,

I have an  Avro generated class (e.g., AvroGenerateClass) and I am using Encoders.bean to get a typed dataset (e.g., Dataset<AvroGeneratedClass>):

Encoder<AvroGeneratedClass> encoder = Encoders.bean(AvroGenereatedClass.class);

Dataset<AvroGeneratedClass> ds = sparkSession.read().parquet(filename).as(encoder);

I am getting an exception from the Encoders.bean call:

"java.lang.UnsupportedOperationException: Cannot have circular references in bean class, but got the circular reference of class class org.apache.avro.Schema"

How can I get a typed dataset from Avro generated classes?

Thanks.

--

Joaquín

[Elkhan Dadashov]

Hi Spark users,

Did anyone resolve this issue?

Encoder<AvroGeneratedClass> encoder = Encoders.bean(AvroGenereatedClass.class);
Dataset<AvroGeneratedClass> ds = sparkSession.read().parquet(filename).as(encoder);

I'm also facing the same problem: "Cannot have circular references in bean class, but got the circular reference of class class org.apache.avro.Schema" 

This happens due to getSchema() method in a generated Avro Java class.

How can I get a typed dataset from Avro generated classes? 

Thanks.

On Wed, Sep 27, 2017 at 3:23 AM Joaquin Tarraga <[hidden email]> wrote:

Hi all,

I have an  Avro generated class (e.g., AvroGenerateClass) and I am using Encoders.bean to get a typed dataset (e.g., Dataset<AvroGeneratedClass>):

Encoder<AvroGeneratedClass> encoder = Encoders.bean(AvroGenereatedClass.class);

Dataset<AvroGeneratedClass> ds = sparkSession.read().parquet(filename).as(encoder);

I am getting an exception from the Encoders.bean call:

"java.lang.UnsupportedOperationException: Cannot have circular references in bean class, but got the circular reference of class class org.apache.avro.Schema"

How can I get a typed dataset from Avro generated classes?

Thanks.

--

Joaquín

--