java.lang.ClassNotFoundException: org.apache.spark.streaming.StreamingContext

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

java.lang.ClassNotFoundException: org.apache.spark.streaming.StreamingContext

Kal El
I am trying to run this code on spark 0.9.0 and I am getting this error: "java.lang.ClassNotFoundException: org.apache.spark.streaming.StreamingContext"

My CLASSPATH variable contains the following path"....spark-0.9.0/assembly/target/scala-2.10/spark-assembly-0.9.0-incubating-hadoop1.0.4.jar"
what else do I need to add here ?

The code works fine in the spark-shell.

"import org.apache.spark.streaming.{Time, Seconds, StreamingContext}
import org.apache.spark.streaming.StreamingContext._

object HdfsWordCount {
  def main(args: Array[String]) {
    if (args.length < 2) {
      System.err.println("Usage: HdfsWordCount <master> <directory>")
      System.exit(1)
    }

  //  StreamingExamples.setStreamingLogLevels()

    // Create the context
//    val ssc = new StreamingContext(args(0), "HdfsWordCount", Seconds(2), System.getenv("SPARK_HOME"), Seq(System.getenv("SPARK_HOME")))
    val ssc = new StreamingContext(args(0), "HdfsWordCount", Seconds(2))
    // Create the FileInputDStream on the directory and use the
    // stream to count words in new files created
    val lines = ssc.textFileStream(args(1))
    val words = lines.flatMap(_.split(" "))
    val wordCounts = words.map(x => (x, 1)).reduceByKey(_ + _)
    wordCounts.print()
    ssc.start()
    ssc.awaitTermination()
  }
}"

Thanks


Reply | Threaded
Open this post in threaded view
|

Re: java.lang.ClassNotFoundException: org.apache.spark.streaming.StreamingContext

Amit Behera
Hi Kal El,
to run you an application outside the spark-shell you need to write a build file (example.sbt) parallel to your source code.
step1: example.sbt

name := "Simple Project"
version := "1.0"
scalaVersion := "2.9.3"
libraryDependencies += "org.apache.spark" %% "spark-­streaming" % "0.9.0­-incubating" 
resolvers ++= Seq( "Akka Repository" at "http://repo.akka.io/releases/","Spray Repository" at "http://repo.spray.cc/")
step2:
download sbt from http://scalasbt.artifactoryonline.com/scalasbt/sbt-native-packages/org/scala-sbt/sbt/0.12.2/sbt.tgz
unzip it and set the path. (up to sbt/bin)
step3:
parallel to your .sbt file type command
$sbt package
$sbt run





On Tue, Feb 11, 2014 at 2:48 PM, Kal El <[hidden email]> wrote:
I am trying to run this code on spark 0.9.0 and I am getting this error: "java.lang.ClassNotFoundException: org.apache.spark.streaming.StreamingContext"

My CLASSPATH variable contains the following path"....spark-0.9.0/assembly/target/scala-2.10/spark-assembly-0.9.0-incubating-hadoop1.0.4.jar"
what else do I need to add here ?

The code works fine in the spark-shell.

"import org.apache.spark.streaming.{Time, Seconds, StreamingContext}
import org.apache.spark.streaming.StreamingContext._

object HdfsWordCount {
  def main(args: Array[String]) {
    if (args.length < 2) {
      System.err.println("Usage: HdfsWordCount <master> <directory>")
      System.exit(1)
    }

  //  StreamingExamples.setStreamingLogLevels()

    // Create the context
//    val ssc = new StreamingContext(args(0), "HdfsWordCount", Seconds(2), System.getenv("SPARK_HOME"), Seq(System.getenv("SPARK_HOME")))
    val ssc = new StreamingContext(args(0), "HdfsWordCount", Seconds(2))
    // Create the FileInputDStream on the directory and use the
    // stream to count words in new files created
    val lines = ssc.textFileStream(args(1))
    val words = lines.flatMap(_.split(" "))
    val wordCounts = words.map(x => (x, 1)).reduceByKey(_ + _)
    wordCounts.print()
    ssc.start()
    ssc.awaitTermination()
  }
}"

Thanks