Typed datataset from Avro generated classes?

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Typed datataset from Avro generated classes?

Joaquin Tarraga
Hi all,
I have an  Avro generated class (e.g., AvroGenerateClass) and I am using Encoders.bean to get a typed dataset (e.g., Dataset<AvroGeneratedClass>):
Encoder<AvroGeneratedClass> encoder = Encoders.bean(AvroGenereatedClass.class);
Dataset<AvroGeneratedClass> ds = sparkSession.read().parquet(filename).as(encoder);
I am getting an exception from the Encoders.bean call:
"java.lang.UnsupportedOperationException: Cannot have circular references in bean class, but got the circular reference of class class org.apache.avro.Schema"

How can I get a typed dataset from Avro generated classes?

Thanks.
--
Joaquín

Reply | Threaded
Open this post in threaded view
|

Re: Typed datataset from Avro generated classes?

Elkhan Dadashov
Hi Spark users,

Did anyone resolve this issue?

Encoder<AvroGeneratedClass> encoder = Encoders.bean(AvroGenereatedClass.class);
Dataset<AvroGeneratedClass> ds = sparkSession.read().parquet(filename).as(encoder);

I'm also facing the same problem: "Cannot have circular references in bean class, but got the circular reference of class class org.apache.avro.Schema" 

This happens due to getSchema() method in a generated Avro Java class.

How can I get a typed dataset from Avro generated classes? 

Thanks.

On Wed, Sep 27, 2017 at 3:23 AM Joaquin Tarraga <[hidden email]> wrote:
Hi all,
I have an  Avro generated class (e.g., AvroGenerateClass) and I am using Encoders.bean to get a typed dataset (e.g., Dataset<AvroGeneratedClass>):
Encoder<AvroGeneratedClass> encoder = Encoders.bean(AvroGenereatedClass.class);
Dataset<AvroGeneratedClass> ds = sparkSession.read().parquet(filename).as(encoder);
I am getting an exception from the Encoders.bean call:
"java.lang.UnsupportedOperationException: Cannot have circular references in bean class, but got the circular reference of class class org.apache.avro.Schema"

How can I get a typed dataset from Avro generated classes?

Thanks.
--
Joaquín



--

Best regards,
Elkhan Dadashov