Issue during Spark streaming with ZeroMQ source

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Issue during Spark streaming with ZeroMQ source

Francis.Hu

Hi, all

 

I installed spark-0.9.1 and zeromq 4.0.1 , and then run below example:

 

./bin/run-example org.apache.spark.streaming.examples.SimpleZeroMQPublisher tcp://127.0.1.1:1234 foo.bar`

./bin/run-example org.apache.spark.streaming.examples.ZeroMQWordCount local[2] tcp://127.0.1.1:1234 foo`

 

No any message was received in ZeroMQWordCount side.

 

Does anyone know what the issue is ?

 

 

Thanks,

Francis

 

Reply | Threaded
Open this post in threaded view
|

Re: Issue during Spark streaming with ZeroMQ source

Prashant Sharma
Unfortunately zeromq 4.0.1 is not supported. https://github.com/apache/spark/blob/master/examples/src/main/scala/org/apache/spark/streaming/examples/ZeroMQWordCount.scala#L63 Says about the version. You will need that version of zeromq to see it work. Basically I have seen it working nicely with zeromq 2.2.0 and if you have jzmq libraries installed performance is much better.

Prashant Sharma


On Tue, Apr 29, 2014 at 12:29 PM, Francis.Hu <[hidden email]> wrote:

Hi, all

 

I installed spark-0.9.1 and zeromq 4.0.1 , and then run below example:

 

./bin/run-example org.apache.spark.streaming.examples.SimpleZeroMQPublisher tcp://127.0.1.1:1234 foo.bar`

./bin/run-example org.apache.spark.streaming.examples.ZeroMQWordCount local[2] tcp://127.0.1.1:1234 foo`

 

No any message was received in ZeroMQWordCount side.

 

Does anyone know what the issue is ?

 

 

Thanks,

Francis

 


Reply | Threaded
Open this post in threaded view
|

答复: Issue during Spark streaming with ZeroMQ source

Francis.Hu

Thanks, Prashant Sharma

 

 

It works right now after degrade zeromq from 4.0.1 to  2.2.

Do you know the new release of spark  whether it will upgrade zeromq ?

Many of our programs are using zeromq 4.0.1, so if in next release ,spark streaming can release with a newer zeromq  that would be better for us.

 

 

Francis.

 

发件人: Prashant Sharma [mailto:[hidden email]]
发送时间: Tuesday, April 29, 2014 15:53
收件人: [hidden email]
主题: Re: Issue during Spark streaming with ZeroMQ source

 

Unfortunately zeromq 4.0.1 is not supported. https://github.com/apache/spark/blob/master/examples/src/main/scala/org/apache/spark/streaming/examples/ZeroMQWordCount.scala#L63 Says about the version. You will need that version of zeromq to see it work. Basically I have seen it working nicely with zeromq 2.2.0 and if you have jzmq libraries installed performance is much better.


Prashant Sharma

 

On Tue, Apr 29, 2014 at 12:29 PM, Francis.Hu <[hidden email]> wrote:

Hi, all

 

I installed spark-0.9.1 and zeromq 4.0.1 , and then run below example:

 

./bin/run-example org.apache.spark.streaming.examples.SimpleZeroMQPublisher tcp://127.0.1.1:1234 foo.bar`

./bin/run-example org.apache.spark.streaming.examples.ZeroMQWordCount local[2] tcp://127.0.1.1:1234 foo`

 

No any message was received in ZeroMQWordCount side.

 

Does anyone know what the issue is ?

 

 

Thanks,

Francis

 

 

Reply | Threaded
Open this post in threaded view
|

Re: 答复: Issue during Spark streaming with ZeroMQ source

Prashant Sharma
Well that is not going to be easy, simply because we depend on akka-zeromq for zeromq support. And since akka does not support the latest zeromq library yet, I doubt if there is something simple that can be done to support it.   

Prashant Sharma


On Tue, Apr 29, 2014 at 2:44 PM, Francis.Hu <[hidden email]> wrote:

Thanks, Prashant Sharma

 

 

It works right now after degrade zeromq from 4.0.1 to  2.2.

Do you know the new release of spark  whether it will upgrade zeromq ?

Many of our programs are using zeromq 4.0.1, so if in next release ,spark streaming can release with a newer zeromq  that would be better for us.

 

 

Francis.

 

发件人: Prashant Sharma [mailto:[hidden email]]
发送时间: Tuesday, April 29, 2014 15:53
收件人: [hidden email]
主题: Re: Issue during Spark streaming with ZeroMQ source

 

Unfortunately zeromq 4.0.1 is not supported. https://github.com/apache/spark/blob/master/examples/src/main/scala/org/apache/spark/streaming/examples/ZeroMQWordCount.scala#L63 Says about the version. You will need that version of zeromq to see it work. Basically I have seen it working nicely with zeromq 2.2.0 and if you have jzmq libraries installed performance is much better.


Prashant Sharma

 

On Tue, Apr 29, 2014 at 12:29 PM, Francis.Hu <[hidden email]> wrote:

Hi, all

 

I installed spark-0.9.1 and zeromq 4.0.1 , and then run below example:

 

./bin/run-example org.apache.spark.streaming.examples.SimpleZeroMQPublisher tcp://127.0.1.1:1234 foo.bar`

./bin/run-example org.apache.spark.streaming.examples.ZeroMQWordCount local[2] tcp://127.0.1.1:1234 foo`

 

No any message was received in ZeroMQWordCount side.

 

Does anyone know what the issue is ?

 

 

Thanks,

Francis