Setting serializer in Spark shell

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Setting serializer in Spark shell

David Thomas
How can I set the default serializer to Kyro when using Spark Shell?
Reply | Threaded
Open this post in threaded view
|

Re: Setting serializer in Spark shell

suman bharadwaj


On Mon, Feb 17, 2014 at 2:52 AM, David Thomas <[hidden email]> wrote:
How can I set the default serializer to Kyro when using Spark Shell?

Reply | Threaded
Open this post in threaded view
|

Re: Setting serializer in Spark shell

Nan Zhu
In reply to this post by David Thomas
try

OPTIONS="-Dspark.serializer=xxxx" bin/spark-shell

but you cannot use -c option, otherwise it will overwrite your external OPTIONS setup

if you have to set -c , you have to modify your bin/spark-shell a bit, before

for o in "$@"; do

add 

OPTIONS="-Dspark.serializer=xxxx

P.S. I didn’t try by myself, but I just modified something similar, I guess it will work

Best,

-- 
Nan Zhu


On Sunday, February 16, 2014 at 4:22 PM, David Thomas wrote:

How can I set the default serializer to Kyro when using Spark Shell?