problem in using Spark-Cassandra connector

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

problem in using Spark-Cassandra connector

Karunya Padala

 

Hi,

 

I am new to spark.  I  encountered an issue when trying to connect to Cassandra using Spark Cassandra connector. Can anyone help me. Following are the details.

 

1) Following Spark and Cassandra versions I am using on LUbuntu12.0.

i)spark-1.0.2-bin-hadoop2

ii) apache-cassandra-2.0.10

 

2) In the Cassandra, i created a key space, table and inserted some data.

 

3)Following libs are specified when starting the spark-shell.

antworks@INHN1I-DW1804:$ spark-shell --jars /home/antworks/lib-spark-cassandra/apache-cassandra-clientutil-2.0.10.jar,/home/antworks/lib-spark-cassandra/apache-cassandra-thrift-2.0.10.jar,/home/antworks/lib-spark-cassandra/cassandra-driver-core-2.0.2.jar,/home/antworks/lib-spark-cassandra/guava-15.0.jar,/home/antworks/lib-spark-cassandra/joda-convert-1.6.jar,/home/antworks/lib-spark-cassandra/joda-time-2.3.jar,/home/antworks/lib-spark-cassandra/libthrift-0.9.1.jar,/home/antworks/lib-spark-cassandra/spark-cassandra-connector_2.10-1.0.0-rc3.jar

 

4) when running the stmt  val rdd = sc.cassandraTable("EmailKeySpace", "Emails")encountered following issue.

 

My application connecting to Cassandra and immediately disconnecting and throwing java.io.IOException: Table not found: EmailKeySpace.Emails

Here is the stack trace.

 

scala> import com.datastax.spark.connector._

import com.datastax.spark.connector._

 

scala> val rdd = sc.cassandraTable("EmailKeySpace", "Emails")

14/09/11 23:06:51 WARN FrameCompressor: Cannot find LZ4 class, you should make sure the LZ4 library is in the classpath if you intend to use it. LZ4 compression will not be available for the protocol.

14/09/11 23:06:51 INFO Cluster: New Cassandra host /172.23.1.68:9042 added

14/09/11 23:06:51 INFO CassandraConnector: Connected to Cassandra cluster: AWCluster

14/09/11 23:06:52 INFO CassandraConnector: Disconnected from Cassandra cluster: AWCluster

java.io.IOException: Table not found: EmailKeySpace.Emails

        at com.datastax.spark.connector.rdd.CassandraRDD.tableDef$lzycompute(CassandraRDD.scala:208)

        at com.datastax.spark.connector.rdd.CassandraRDD.tableDef(CassandraRDD.scala:205)

        at com.datastax.spark.connector.rdd.CassandraRDD.<init>(CassandraRDD.scala:212)

        at com.datastax.spark.connector.SparkContextFunctions.cassandraTable(SparkContextFunctions.scala:48)

        at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:15)

        at $iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:20)

        at $iwC$$iwC$$iwC$$iwC.<init>(<console>:22)

        at $iwC$$iwC$$iwC.<init>(<console>:24)

        at $iwC$$iwC.<init>(<console>:26)

        at $iwC.<init>(<console>:28)

        at <init>(<console>:30)

        at .<init>(<console>:34)

        at .<clinit>(<console>)

        at .<init>(<console>:7)

        at .<clinit>(<console>)

        at $print(<console>)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:606)

        at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:788)

        at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1056)

        at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:614)

        at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:645)

        at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:609)

        at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:796)

        at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:841)

        at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:753)

        at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:601)

        at org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:608)

        at org.apache.spark.repl.SparkILoop.loop(SparkILoop.scala:611)

        at org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply$mcZ$sp(SparkILoop.scala:936)

        at org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply(SparkILoop.scala:884)

        at org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply(SparkILoop.scala:884)

        at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)

        at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:884)

        at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:982)

        at org.apache.spark.repl.Main$.main(Main.scala:31)

        at org.apache.spark.repl.Main.main(Main.scala)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:606)

        at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:303)

        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:55)

        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

 

 

 




DISCLAIMER:

This email may contain confidential information and is intended only for the use of the specific individual(s) to which it is addressed. If you are not the intended recipient of this email, you are hereby notified that any unauthorized use, dissemination or copying of this email or the information contained in it or attached to it is strictly prohibited. If you received this message in error, please immediately notify the sender at Cyient and delete the original message.
Reply | Threaded
Open this post in threaded view
|

Re: problem in using Spark-Cassandra connector

reddyraja
You will have to create create KeySpace and Table.
See the message,
Table not found: EmailKeySpace.Emails

Looks like you have not created the Emails table.


On Thu, Sep 11, 2014 at 6:04 PM, Karunya Padala <[hidden email]> wrote:

 

Hi,

 

I am new to spark.  I  encountered an issue when trying to connect to Cassandra using Spark Cassandra connector. Can anyone help me. Following are the details.

 

1) Following Spark and Cassandra versions I am using on LUbuntu12.0.

i)spark-1.0.2-bin-hadoop2

ii) apache-cassandra-2.0.10

 

2) In the Cassandra, i created a key space, table and inserted some data.

 

3)Following libs are specified when starting the spark-shell.

antworks@INHN1I-DW1804:$ spark-shell --jars /home/antworks/lib-spark-cassandra/apache-cassandra-clientutil-2.0.10.jar,/home/antworks/lib-spark-cassandra/apache-cassandra-thrift-2.0.10.jar,/home/antworks/lib-spark-cassandra/cassandra-driver-core-2.0.2.jar,/home/antworks/lib-spark-cassandra/guava-15.0.jar,/home/antworks/lib-spark-cassandra/joda-convert-1.6.jar,/home/antworks/lib-spark-cassandra/joda-time-2.3.jar,/home/antworks/lib-spark-cassandra/libthrift-0.9.1.jar,/home/antworks/lib-spark-cassandra/spark-cassandra-connector_2.10-1.0.0-rc3.jar

 

4) when running the stmt  val rdd = sc.cassandraTable("EmailKeySpace", "Emails")encountered following issue.

 

My application connecting to Cassandra and immediately disconnecting and throwing java.io.IOException: Table not found: EmailKeySpace.Emails

Here is the stack trace.

 

scala> import com.datastax.spark.connector._

import com.datastax.spark.connector._

 

scala> val rdd = sc.cassandraTable("EmailKeySpace", "Emails")

14/09/11 23:06:51 WARN FrameCompressor: Cannot find LZ4 class, you should make sure the LZ4 library is in the classpath if you intend to use it. LZ4 compression will not be available for the protocol.

14/09/11 23:06:51 INFO Cluster: New Cassandra host /172.23.1.68:9042 added

14/09/11 23:06:51 INFO CassandraConnector: Connected to Cassandra cluster: AWCluster

14/09/11 23:06:52 INFO CassandraConnector: Disconnected from Cassandra cluster: AWCluster

java.io.IOException: Table not found: EmailKeySpace.Emails

        at com.datastax.spark.connector.rdd.CassandraRDD.tableDef$lzycompute(CassandraRDD.scala:208)

        at com.datastax.spark.connector.rdd.CassandraRDD.tableDef(CassandraRDD.scala:205)

        at com.datastax.spark.connector.rdd.CassandraRDD.<init>(CassandraRDD.scala:212)

        at com.datastax.spark.connector.SparkContextFunctions.cassandraTable(SparkContextFunctions.scala:48)

        at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:15)

        at $iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:20)

        at $iwC$$iwC$$iwC$$iwC.<init>(<console>:22)

        at $iwC$$iwC$$iwC.<init>(<console>:24)

        at $iwC$$iwC.<init>(<console>:26)

        at $iwC.<init>(<console>:28)

        at <init>(<console>:30)

        at .<init>(<console>:34)

        at .<clinit>(<console>)

        at .<init>(<console>:7)

        at .<clinit>(<console>)

        at $print(<console>)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:606)

        at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:788)

        at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1056)

        at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:614)

        at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:645)

        at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:609)

        at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:796)

        at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:841)

        at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:753)

        at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:601)

        at org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:608)

        at org.apache.spark.repl.SparkILoop.loop(SparkILoop.scala:611)

        at org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply$mcZ$sp(SparkILoop.scala:936)

        at org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply(SparkILoop.scala:884)

        at org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply(SparkILoop.scala:884)

        at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)

        at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:884)

        at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:982)

        at org.apache.spark.repl.Main$.main(Main.scala:31)

        at org.apache.spark.repl.Main.main(Main.scala)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:606)

        at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:303)

        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:55)

        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

 

 

 




DISCLAIMER:

This email may contain confidential information and is intended only for the use of the specific individual(s) to which it is addressed. If you are not the intended recipient of this email, you are hereby notified that any unauthorized use, dissemination or copying of this email or the information contained in it or attached to it is strictly prohibited. If you received this message in error, please immediately notify the sender at Cyient and delete the original message.



--
"In this world, you either have an excuse or a story. I preferred to have a story"
Reply | Threaded
Open this post in threaded view
|

RE: problem in using Spark-Cassandra connector

Karunya Padala

I have created key space called EmailKeySpace’and table called Emails and inserted some data in the Cassandra. See my Cassandra console screen shot.

 

 

 

 

Regards,

Karunya.

 

From: Reddy Raja [mailto:[hidden email]]
Sent: 11 September 2014 18:07
To: Karunya Padala
Cc: [hidden email]
Subject: Re: problem in using Spark-Cassandra connector

 

You will have to create create KeySpace and Table.

See the message,

Table not found: EmailKeySpace.Emails

 

Looks like you have not created the Emails table.

 

 

On Thu, Sep 11, 2014 at 6:04 PM, Karunya Padala <[hidden email]> wrote:

 

Hi,

 

I am new to spark.  I  encountered an issue when trying to connect to Cassandra using Spark Cassandra connector. Can anyone help me. Following are the details.

 

1) Following Spark and Cassandra versions I am using on LUbuntu12.0.

i)spark-1.0.2-bin-hadoop2

ii) apache-cassandra-2.0.10

 

2) In the Cassandra, i created a key space, table and inserted some data.

 

3)Following libs are specified when starting the spark-shell.

antworks@INHN1I-DW1804:$ spark-shell --jars /home/antworks/lib-spark-cassandra/apache-cassandra-clientutil-2.0.10.jar,/home/antworks/lib-spark-cassandra/apache-cassandra-thrift-2.0.10.jar,/home/antworks/lib-spark-cassandra/cassandra-driver-core-2.0.2.jar,/home/antworks/lib-spark-cassandra/guava-15.0.jar,/home/antworks/lib-spark-cassandra/joda-convert-1.6.jar,/home/antworks/lib-spark-cassandra/joda-time-2.3.jar,/home/antworks/lib-spark-cassandra/libthrift-0.9.1.jar,/home/antworks/lib-spark-cassandra/spark-cassandra-connector_2.10-1.0.0-rc3.jar

 

4) when running the stmt  val rdd = sc.cassandraTable("EmailKeySpace", "Emails")encountered following issue.

 

My application connecting to Cassandra and immediately disconnecting and throwing java.io.IOException: Table not found: EmailKeySpace.Emails

Here is the stack trace.

 

scala> import com.datastax.spark.connector._

import com.datastax.spark.connector._

 

scala> val rdd = sc.cassandraTable("EmailKeySpace", "Emails")

14/09/11 23:06:51 WARN FrameCompressor: Cannot find LZ4 class, you should make sure the LZ4 library is in the classpath if you intend to use it. LZ4 compression will not be available for the protocol.

14/09/11 23:06:51 INFO Cluster: New Cassandra host /172.23.1.68:9042 added

14/09/11 23:06:51 INFO CassandraConnector: Connected to Cassandra cluster: AWCluster

14/09/11 23:06:52 INFO CassandraConnector: Disconnected from Cassandra cluster: AWCluster

java.io.IOException: Table not found: EmailKeySpace.Emails

        at com.datastax.spark.connector.rdd.CassandraRDD.tableDef$lzycompute(CassandraRDD.scala:208)

        at com.datastax.spark.connector.rdd.CassandraRDD.tableDef(CassandraRDD.scala:205)

        at com.datastax.spark.connector.rdd.CassandraRDD.<init>(CassandraRDD.scala:212)

        at com.datastax.spark.connector.SparkContextFunctions.cassandraTable(SparkContextFunctions.scala:48)

        at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:15)

        at $iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:20)

        at $iwC$$iwC$$iwC$$iwC.<init>(<console>:22)

        at $iwC$$iwC$$iwC.<init>(<console>:24)

        at $iwC$$iwC.<init>(<console>:26)

        at $iwC.<init>(<console>:28)

        at <init>(<console>:30)

        at .<init>(<console>:34)

        at .<clinit>(<console>)

        at .<init>(<console>:7)

        at .<clinit>(<console>)

        at $print(<console>)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:606)

        at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:788)

        at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1056)

        at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:614)

        at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:645)

        at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:609)

        at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:796)

        at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:841)

        at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:753)

        at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:601)

        at org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:608)

        at org.apache.spark.repl.SparkILoop.loop(SparkILoop.scala:611)

        at org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply$mcZ$sp(SparkILoop.scala:936)

        at org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply(SparkILoop.scala:884)

        at org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply(SparkILoop.scala:884)

        at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)

        at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:884)

        at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:982)

        at org.apache.spark.repl.Main$.main(Main.scala:31)

        at org.apache.spark.repl.Main.main(Main.scala)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:606)

        at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:303)

        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:55)

        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

 

 

 

 



DISCLAIMER:

This email may contain confidential information and is intended only for the use of the specific individual(s) to which it is addressed. If you are not the intended recipient of this email, you are hereby notified that any unauthorized use, dissemination or copying of this email or the information contained in it or attached to it is strictly prohibited. If you received this message in error, please immediately notify the sender at Cyient and delete the original message.



 

--
"In this world, you either have an excuse or a story. I preferred to have a story"