Encoder for JValue

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Encoder for JValue

Arko Provo Mukherjee
Hello Spark Gurus,

I am running into an issue with Encoding and wanted your help.

I have a case class with a JObject in it. Ex:
case class SomeClass(a: String, b: JObject)

I also have an encoder for this case class:
val encoder = Encoders.product[SomeClass]

Now I am creating a DataFrame with the tuple (a, b) from my transformations and converting into a DataSet: 
df.as[SomeClass](encoder)

When I do this, I get the following error:
java.lang.UnsupportedOperationException: No Encoder found for org.json4s.JsonAST.JValue

Appreciate any help regarding this issue. 

Many thanks in advance!
Warm regards
Arko


Reply | Threaded
Open this post in threaded view
|

Re: Encoder for JValue

muthu
A naive workaround may be to transform the json4s JValue to String (using something like compact()) and process it as String? Once you are done with the last action, you could write it back as JValue (using something like parse())

Thanks,
Muthu

On Wed, Sep 19, 2018 at 6:35 AM Arko Provo Mukherjee <[hidden email]> wrote:
Hello Spark Gurus,

I am running into an issue with Encoding and wanted your help.

I have a case class with a JObject in it. Ex:
case class SomeClass(a: String, b: JObject)

I also have an encoder for this case class:
val encoder = Encoders.product[SomeClass]

Now I am creating a DataFrame with the tuple (a, b) from my transformations and converting into a DataSet: 
df.as[SomeClass](encoder)

When I do this, I get the following error:
java.lang.UnsupportedOperationException: No Encoder found for org.json4s.JsonAST.JValue

Appreciate any help regarding this issue. 

Many thanks in advance!
Warm regards
Arko


Reply | Threaded
Open this post in threaded view
|

Re: Encoder for JValue

Arko Provo Mukherjee
Hello Muthu,

Many thanks for your reply. That is what we are currently doing. 

However, we finally load the data somewhere and we need to have JSON objects rather than serialized strings.

Hence I was wondering if there are encoders our there for JObject and if I can somehow pass that information to Spark.

Thanks & regards
Arko


On Tue, Sep 18, 2018 at 11:39 PM Muthu Jayakumar <[hidden email]> wrote:
A naive workaround may be to transform the json4s JValue to String (using something like compact()) and process it as String? Once you are done with the last action, you could write it back as JValue (using something like parse())

Thanks,
Muthu

On Wed, Sep 19, 2018 at 6:35 AM Arko Provo Mukherjee <[hidden email]> wrote:
Hello Spark Gurus,

I am running into an issue with Encoding and wanted your help.

I have a case class with a JObject in it. Ex:
case class SomeClass(a: String, b: JObject)

I also have an encoder for this case class:
val encoder = Encoders.product[SomeClass]

Now I am creating a DataFrame with the tuple (a, b) from my transformations and converting into a DataSet: 
df.as[SomeClass](encoder)

When I do this, I get the following error:
java.lang.UnsupportedOperationException: No Encoder found for org.json4s.JsonAST.JValue

Appreciate any help regarding this issue. 

Many thanks in advance!
Warm regards
Arko