Spark-cassandra-connector 1.0.0-rc5: java.io.NotSerializableException

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Spark-cassandra-connector 1.0.0-rc5: java.io.NotSerializableException

Shing Hing Man-2

Hi,

My version of Spark is 1.0.2.
I am trying to use Spark-cassandra-connector to execute an update csql statement inside
an CassandraConnector(conf).withSessionDo  block :

CassandraConnector(conf).withSessionDo {
      session =>
        {
          myRdd.foreach {
            case (ip, values) =>            
              session.execute( {An csql update statement})
          }

        }

    }
   
 The above code is in the main method of the driver object.
 When I run the above code (in local mode), I get an exception :
 
  java.io.NotSerializableException: com.datastax.spark.connector.cql.SessionProxy
 
 Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task not serializable: java.io.NotSerializableException: com.datastax.spark.connector.cql.SessionProxy
    at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1049)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1033)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1031)
    at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
    at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1031)
    at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$submitMissingTasks(DAGScheduler.scala:772)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskCompletion$16$$anonfun$apply$1.apply$mcVI$sp(DAGScheduler.scala:906)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskCompletion$16$$anonfun$apply$1.apply(DAGScheduler.scala:903)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskCompletion$16$$anonfun$apply$1.apply(DAGScheduler.scala:903)
 
 
Is there a way to use an RDD inside an   CassandraConnector(conf).withSessionDo  block  ?

Thanks in advance for any assistance !
 
Shing

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]