How to confirm serializer type on workers?

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

How to confirm serializer type on workers?

Aureliano Buendia
Hi,


I'm using kryo serializer, but getting java.io.EOFException error when broadcasting an object. It seems one of the reasons for getting this error is that the workers are not aware of the kryo serializer.

In the app code, before creating the context, I have:

System.setProperty("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
System.setProperty("spark.kryo.registrator", "myRegistrator")


Does this only work on the master and not on the workers?

I've also tried setting this in conf/spark-env.sh:

export SPARK_JAVA_OPTS="-Dspark.serializer=org.apache.spark.serializer.KryoSerializer -Dspark.kryo.registrator=my.Registrator"

But I'm still getting the eof error.

How is it possible to programmatically set the kryo serializer on the workers side, the same as it is done on the master?
Reply | Threaded
Open this post in threaded view
|

Re: How to confirm serializer type on workers?

Eugen Cepoi
Do you have the stacktrace?
I had something similar, where the Kryo deser was throwing EOF, but in fact EOF means nothing, spark catches Kryo exceptions and then throws EOF (and loses the reason...), in my case kryo couldn't find the class to which to deser.


2014/1/8 Aureliano Buendia <[hidden email]>
Hi,


I'm using kryo serializer, but getting java.io.EOFException error when broadcasting an object. It seems one of the reasons for getting this error is that the workers are not aware of the kryo serializer.

In the app code, before creating the context, I have:

System.setProperty("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
System.setProperty("spark.kryo.registrator", "myRegistrator")


Does this only work on the master and not on the workers?

I've also tried setting this in conf/spark-env.sh:

export SPARK_JAVA_OPTS="-Dspark.serializer=org.apache.spark.serializer.KryoSerializer -Dspark.kryo.registrator=my.Registrator"

But I'm still getting the eof error.

How is it possible to programmatically set the kryo serializer on the workers side, the same as it is done on the master?