Spark Release 0.9.0 missing org.apache.spark.streaming package + misleading documentation on http://spark.incubator.apache.org/releases/spark-release-0-9-0.html

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Spark Release 0.9.0 missing org.apache.spark.streaming package + misleading documentation on http://spark.incubator.apache.org/releases/spark-release-0-9-0.html

amoc

I pull l the following version of spark via sbt

    "org.apache.spark" % "spark-core_2.10" % "0.9.0-incubating"

Which according to the green button at the bottom of this page http://spark.incubator.apache.org/streaming/ should have streaming.

 

 

On this page: http://spark.incubator.apache.org/releases/spark-release-0-9-0.html

It talks about improvements to spark:

A few aspects of the API have been improved:

  • DStream and PairDStream classes have been moved from org.apache.spark.streaming to org.apache.spark.streaming.dstream to keep it consistent with org.apache.spark.rdd.RDD.
  • DStream.foreach has been renamed to foreachRDD to make it explicit that it works for every RDD, not every element”

These package don’t exist in spark anymore. See my attachment. Where did it go? I also can’t find org.apache.spark.streaming.StreamingContext

 

 

-Adrian

 


missing.streaming.pkg.png (205K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Spark Release 0.9.0 missing org.apache.spark.streaming package + misleading documentation on http://spark.incubator.apache.org/releases/spark-release-0-9-0.html

Evan R. Sparks
You'll need to link against the spark-streaming artifact as well - see the linking section here:


On Wed, Feb 12, 2014 at 2:43 PM, Adrian Mocanu <[hidden email]> wrote:

I pull l the following version of spark via sbt

    "org.apache.spark" % "spark-core_2.10" % "0.9.0-incubating"

Which according to the green button at the bottom of this page http://spark.incubator.apache.org/streaming/ should have streaming.

 

 

On this page: http://spark.incubator.apache.org/releases/spark-release-0-9-0.html

It talks about improvements to spark:

A few aspects of the API have been improved:

  • DStream and PairDStream classes have been moved from org.apache.spark.streaming to org.apache.spark.streaming.dstream to keep it consistent with org.apache.spark.rdd.RDD.
  • DStream.foreach has been renamed to foreachRDD to make it explicit that it works for every RDD, not every element”

These package don’t exist in spark anymore. See my attachment. Where did it go? I also can’t find org.apache.spark.streaming.StreamingContext

 

 

-Adrian

 


Reply | Threaded
Open this post in threaded view
|

Re: Spark Release 0.9.0 missing org.apache.spark.streaming package + misleading documentation on http://spark.incubator.apache.org/releases/spark-release-0-9-0.html

Aaron Davidson
Be sure to link against the 0.9.0 version, though! Here are the updated docs:



On Wed, Feb 12, 2014 at 2:48 PM, Evan R. Sparks <[hidden email]> wrote:
You'll need to link against the spark-streaming artifact as well - see the linking section here:


On Wed, Feb 12, 2014 at 2:43 PM, Adrian Mocanu <[hidden email]> wrote:

I pull l the following version of spark via sbt

    "org.apache.spark" % "spark-core_2.10" % "0.9.0-incubating"

Which according to the green button at the bottom of this page http://spark.incubator.apache.org/streaming/ should have streaming.

 

 

On this page: http://spark.incubator.apache.org/releases/spark-release-0-9-0.html

It talks about improvements to spark:

A few aspects of the API have been improved:

  • DStream and PairDStream classes have been moved from org.apache.spark.streaming to org.apache.spark.streaming.dstream to keep it consistent with org.apache.spark.rdd.RDD.
  • DStream.foreach has been renamed to foreachRDD to make it explicit that it works for every RDD, not every element”

These package don’t exist in spark anymore. See my attachment. Where did it go? I also can’t find org.apache.spark.streaming.StreamingContext

 

 

-Adrian

 



Reply | Threaded
Open this post in threaded view
|

Re: Spark Release 0.9.0 missing org.apache.spark.streaming package + misleading documentation on http://spark.incubator.apache.org/releases/spark-release-0-9-0.html

Mark Hamstra
In reply to this post by amoc
These package don’t exist in spark anymore.

They never did exist in spark-core.  If your project depends on spark-streaming, then it should pull in the spark-core dependency as well.  If you only depend on spark-core, then you won't get streaming if you don't also ask for it.  Similarly for other packages within Spark that depend on spark-core.  


On Wed, Feb 12, 2014 at 2:43 PM, Adrian Mocanu <[hidden email]> wrote:

I pull l the following version of spark via sbt

    "org.apache.spark" % "spark-core_2.10" % "0.9.0-incubating"

Which according to the green button at the bottom of this page http://spark.incubator.apache.org/streaming/ should have streaming.

 

 

On this page: http://spark.incubator.apache.org/releases/spark-release-0-9-0.html

It talks about improvements to spark:

A few aspects of the API have been improved:

  • DStream and PairDStream classes have been moved from org.apache.spark.streaming to org.apache.spark.streaming.dstream to keep it consistent with org.apache.spark.rdd.RDD.
  • DStream.foreach has been renamed to foreachRDD to make it explicit that it works for every RDD, not every element”

These package don’t exist in spark anymore. See my attachment. Where did it go? I also can’t find org.apache.spark.streaming.StreamingContext

 

 

-Adrian

 


Reply | Threaded
Open this post in threaded view
|

Re: Spark Release 0.9.0 missing org.apache.spark.streaming package + misleading documentation on http://spark.incubator.apache.org/releases/spark-release-0-9-0.html

Evan R. Sparks
In reply to this post by Aaron Davidson
Ah, good catch Aaron - my bad!


On Wed, Feb 12, 2014 at 2:50 PM, Aaron Davidson <[hidden email]> wrote:
Be sure to link against the 0.9.0 version, though! Here are the updated docs:



On Wed, Feb 12, 2014 at 2:48 PM, Evan R. Sparks <[hidden email]> wrote:
You'll need to link against the spark-streaming artifact as well - see the linking section here:


On Wed, Feb 12, 2014 at 2:43 PM, Adrian Mocanu <[hidden email]> wrote:

I pull l the following version of spark via sbt

    "org.apache.spark" % "spark-core_2.10" % "0.9.0-incubating"

Which according to the green button at the bottom of this page http://spark.incubator.apache.org/streaming/ should have streaming.

 

 

On this page: http://spark.incubator.apache.org/releases/spark-release-0-9-0.html

It talks about improvements to spark:

A few aspects of the API have been improved:

  • DStream and PairDStream classes have been moved from org.apache.spark.streaming to org.apache.spark.streaming.dstream to keep it consistent with org.apache.spark.rdd.RDD.
  • DStream.foreach has been renamed to foreachRDD to make it explicit that it works for every RDD, not every element”

These package don’t exist in spark anymore. See my attachment. Where did it go? I also can’t find org.apache.spark.streaming.StreamingContext

 

 

-Adrian

 




Reply | Threaded
Open this post in threaded view
|

RE: Spark Release 0.9.0 missing org.apache.spark.streaming package + misleading documentation on http://spark.incubator.apache.org/releases/spark-release-0-9-0.html

amoc

Thank you guys. Including  "org.apache.spark" % "spark-streaming_2.10" % "0.9.0-incubating" worked.

 

My confusion came from the fact that on the page (http://spark.incubator.apache.org/streaming/ ) on the big green button at the bottom it says “Download Spark includes Spark Streaming”

I think it would help to change the text on the button.

 

-A

From: Evan R. Sparks [mailto:[hidden email]]
Sent: February-12-14 6:03 PM
To: [hidden email]
Subject: Re: Spark Release 0.9.0 missing org.apache.spark.streaming package + misleading documentation on http://spark.incubator.apache.org/releases/spark-release-0-9-0.html

 

Ah, good catch Aaron - my bad!

 

On Wed, Feb 12, 2014 at 2:50 PM, Aaron Davidson <[hidden email]> wrote:

Be sure to link against the 0.9.0 version, though! Here are the updated docs:

 

 

On Wed, Feb 12, 2014 at 2:48 PM, Evan R. Sparks <[hidden email]> wrote:

You'll need to link against the spark-streaming artifact as well - see the linking section here:

 

On Wed, Feb 12, 2014 at 2:43 PM, Adrian Mocanu <[hidden email]> wrote:

I pull l the following version of spark via sbt

    "org.apache.spark" % "spark-core_2.10" % "0.9.0-incubating"

Which according to the green button at the bottom of this page http://spark.incubator.apache.org/streaming/ should have streaming.

 

 

On this page: http://spark.incubator.apache.org/releases/spark-release-0-9-0.html

It talks about improvements to spark:

“A few aspects of the API have been improved:

  • DStream and PairDStream classes have been moved from org.apache.spark.streaming to org.apache.spark.streaming.dstream to keep it consistent with org.apache.spark.rdd.RDD.
  • DStream.foreach has been renamed to foreachRDD to make it explicit that it works for every RDD, not every element”

These package don’t exist in spark anymore. See my attachment. Where did it go? I also can’t find org.apache.spark.streaming.StreamingContext

 

 

-Adrian