Case class with POJO - encoder issues

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Case class with POJO - encoder issues

Jason White
This post has NOT been accepted by the mailing list yet.
I'd like to create a Dataset using some classes from Geotools to do some geospatial analysis. In particular, I'm trying to use Spark to distribute the work based on ID and label fields that I extract from the polygon data.

My simplified case class looks like this:
implicit val geometryEncoder: Encoder[Geometry] = Encoders.kryo[Geometry]
case class IndexedGeometry(label: String, tract: Geometry)

When I try to create a dataset using this case class, it give me this error message:
Exception in thread "main" java.lang.UnsupportedOperationException: No Encoder found for com.vividsolutions.jts.geom.Geometry
- field (class: "com.vividsolutions.jts.geom.Geometry", name: "tract")
- root class: "org.me.HelloWorld.IndexedGeometry"

If I add another encoder for my case class...:
implicit val indexedGeometryEncoder: Encoder[IndexedGeometry] = Encoders.kryo[IndexedGeometry]

...it works, but now the entire dataset has a single field, "value", and it's a binary blob.

Is there a way to do what I'm trying to do?
I believe this PR is related, but it's been idle since December: https://github.com/apache/spark/pull/15918
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Case class with POJO - encoder issues

geoHeil
This post has NOT been accepted by the mailing list yet.
http://stackoverflow.com/questions/36648128/how-to-store-custom-objects-in-a-dataset describes the Problem. Actually, I have the same Problem. Is there a simple way to build such an Encoder which serializes into multiple fields? I would not want to replicate the Whole JTS geometry class hierarchy only for spark.
Loading...