Spark HDFS read/write in local mode and cluster mode behaviour

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Spark HDFS read/write in local mode and cluster mode behaviour

Jaggu
Hi Team,
I was trying to run a WordCount app in cluster mode and stand alone mode in a 3 node Spark cluster.
The code reads input from HDFS and writes the results to HDFS.

When I executed it in cluster I am getting this error http://pastebin.com/raw.php?i=Ezw9C3W8 . Any idea how to resolve the same.

Spark cluster and Hadoop cluster is same. I have included the HADOOP_CONF variable in the spark-env file. I can do a passwordless ssh to all the machines in my Spark cluster too.

Best regards

Jagan