reduceByKey() is not a member of org.apache.spark.streaming.dstream.DStream[(String, Int)]

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

reduceByKey() is not a member of org.apache.spark.streaming.dstream.DStream[(String, Int)]

bethesda
This post was updated on .
I am getting this error when trying code (from the following page) in the shell:

http://spark.incubator.apache.org/docs/latest/streaming-programming-guide.html

I believe that DStream[(String, Int)] is precisely the class that should have this function so I am confused.  

Thanks!

scala> val words = lines.flatMap(_.split(" "))
words: org.apache.spark.streaming.dstream.DStream[String] = org.apache.spark.streaming.dstream.FlatMappedDStream@56731b34

scala> val pairs = words.map(word => (word, 1))
pairs: org.apache.spark.streaming.dstream.DStream[(String, Int)] = org.apache.spark.streaming.dstream.MappedDStream@3145888

scala> val wordCounts = pairs.reduceByKey(_ + _)
<console>:26: error: value reduceByKey is not a member of org.apache.spark.streaming.dstream.DStream[(String, Int)]
       val wordCounts = pairs.reduceByKey(_ + _)
Reply | Threaded
Open this post in threaded view
|

Re: reduceByKey() is not a member of org.apache.spark.streaming.dstream.DStream[(String, Int)]

Tathagata Das
You have to "import StreamingContext._ " 
That imports implicit conversions that allow reduceByKey() to be applied on DStreams with key-value pairs.

TD




On Tue, Feb 18, 2014 at 12:03 PM, bethesda <[hidden email]> wrote:
I am getting this error when trying to code from the following page in the
shell:

http://spark.incubator.apache.org/docs/latest/streaming-programming-guide.html

I believe that DStream[(String, Int)] is precisely the class that should
have this function so I am confused.

Thanks!

scala> val words = lines.flatMap(_.split(" "))
words: org.apache.spark.streaming.dstream.DStream[String] =
org.apache.spark.streaming.dstream.FlatMappedDStream@56731b34

scala> val pairs = words.map(word => (word, 1))
pairs: org.apache.spark.streaming.dstream.DStream[(String, Int)] =
org.apache.spark.streaming.dstream.MappedDStream@3145888

scala> val wordCounts = pairs.reduceByKey(_ + _)
<console>:26: error: value reduceByKey is not a member of
org.apache.spark.streaming.dstream.DStream[(String, Int)]
       val wordCounts = pairs.reduceByKey(_ + _)




--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/reduceByKey-is-not-a-member-of-org-apache-spark-streaming-dstream-DStream-String-Int-tp1694.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

Reply | Threaded
Open this post in threaded view
|

Re: reduceByKey() is not a member of org.apache.spark.streaming.dstream.DStream[(String, Int)]

bethesda
Thanks, I'll try it!