network wordcount example

[eric perler]

Hello

i just started working with spark today... and i am trying to run the wordcount network example

i created a socket server and client.. and i am sending data to the server in an infinite loop

when i run the spark class.. i see this output in the console...

-------------------------------------------

Time: 1396281891000 ms

-------------------------------------------

14/03/31 11:04:51 INFO SparkContext: Job finished: take at DStream.scala:586, took 0.056794606 s

14/03/31 11:04:51 INFO JobScheduler: Finished job streaming job 1396281891000 ms.0 from job set of time 1396281891000 ms

14/03/31 11:04:51 INFO JobScheduler: Total delay: 0.101 s for time 1396281891000 ms (execution: 0.058 s)

14/03/31 11:04:51 INFO TaskSchedulerImpl: Remove TaskSet 3.0 from pool 

but i dont see any output from the workcount operation when i make this call...

wordCounts.print();

any help is greatly appreciated

thanks in advance

[Diana Carroll]

Not sure what data you are sending in.  You could try calling "lines.print()" instead which should just output everything that comes in on the stream.  Just to test that your socket is receiving what you think you are sending.

On Mon, Mar 31, 2014 at 12:18 PM, eric perler <[hidden email]> wrote:

Hello

i just started working with spark today... and i am trying to run the wordcount network example

i created a socket server and client.. and i am sending data to the server in an infinite loop

when i run the spark class.. i see this output in the console...

-------------------------------------------

Time: 1396281891000 ms

-------------------------------------------

14/03/31 11:04:51 INFO SparkContext: Job finished: take at DStream.scala:586, took 0.056794606 s

14/03/31 11:04:51 INFO JobScheduler: Finished job streaming job 1396281891000 ms.0 from job set of time 1396281891000 ms

14/03/31 11:04:51 INFO JobScheduler: Total delay: 0.101 s for time 1396281891000 ms (execution: 0.058 s)

14/03/31 11:04:51 INFO TaskSchedulerImpl: Remove TaskSet 3.0 from pool 

but i dont see any output from the workcount operation when i make this call...

wordCounts.print();

any help is greatly appreciated

thanks in advance

[cfregly]

@eric-

i saw this exact issue recently while working on the KinesisWordCount.

are you passing "local[2]" to your example as the MASTER arg versus just "local" or "local[1]"?

you need at least 2.  it's documented as "n>1" in the scala source docs - which is easy to mistake for n>=1.

i just ran the NetworkWordCount sample and confirmed that local[1] does not work, but  local[2] does work.

give that a whirl.

-chris

On Mon, Mar 31, 2014 at 10:41 AM, Diana Carroll <[hidden email]> wrote:

Not sure what data you are sending in.  You could try calling "lines.print()" instead which should just output everything that comes in on the stream.  Just to test that your socket is receiving what you think you are sending.

On Mon, Mar 31, 2014 at 12:18 PM, eric perler <[hidden email]> wrote:

Hello

i just started working with spark today... and i am trying to run the wordcount network example

i created a socket server and client.. and i am sending data to the server in an infinite loop

when i run the spark class.. i see this output in the console...

-------------------------------------------

Time: 1396281891000 ms

-------------------------------------------

14/03/31 11:04:51 INFO SparkContext: Job finished: take at DStream.scala:586, took 0.056794606 s

14/03/31 11:04:51 INFO JobScheduler: Finished job streaming job 1396281891000 ms.0 from job set of time 1396281891000 ms

14/03/31 11:04:51 INFO JobScheduler: Total delay: 0.101 s for time 1396281891000 ms (execution: 0.058 s)

14/03/31 11:04:51 INFO TaskSchedulerImpl: Remove TaskSet 3.0 from pool 

but i dont see any output from the workcount operation when i make this call...

wordCounts.print();

any help is greatly appreciated

thanks in advance