Change print() in JavaNetworkWordCount

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Change print() in JavaNetworkWordCount

eduardocalfaia
Hi Guys,
I think that I already did this question, but I don't remember if anyone
has answered me. I would like changing in the function print() the
quantity of words and the frequency number that are sent to driver's
screen. The default value is 10.

Anyone could help me with this?

Best Regards

--
Informativa sulla Privacy: http://www.unibs.it/node/8155
Reply | Threaded
Open this post in threaded view
|

Re: Change print() in JavaNetworkWordCount

Sourav Chandra
You can extend DStream and override print() method. Then you can create your own DSTream extending from this.


On Tue, Mar 25, 2014 at 6:07 PM, Eduardo Costa Alfaia <[hidden email]> wrote:
Hi Guys,
I think that I already did this question, but I don't remember if anyone has answered me. I would like changing in the function print() the quantity of words and the frequency number that are sent to driver's screen. The default value is 10.

Anyone could help me with this?

Best Regards

--
Informativa sulla Privacy: http://www.unibs.it/node/8155



--

Sourav Chandra

Senior Software Engineer

· · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · ·

[hidden email]

o: +91 80 4121 8723

m: +91 988 699 3746

skype: sourav.chandra

Livestream

"Ajmera Summit", First Floor, #3/D, 68 Ward, 3rd Cross, 7th C Main, 3rd Block, Koramangala Industrial Area,

Bangalore 560034

www.livestream.com

Reply | Threaded
Open this post in threaded view
|

Re: Change print() in JavaNetworkWordCount

eduardocalfaia
Thanks Guys for reply, but I have found this piece of code in streaming:
 def print() {
585     def foreachFunc = (rdd: RDD[T], time: Time) => {
586       //val first11 = rdd.take(11)
587       //val first100 = rdd.take(100)
588         val total = rdd.collect()
589       println ("-------------------------------------------")
590       println ("Time: " + time)
591       println ("-------------------------------------------")
592       //first100.take(100).foreach(println)
593         total.collect().foreach(println)
594       //if (first100.size > 99) println("(...)")
595       println()
596     }
597     new ForEachDStream(this, context.sparkContext.clean(foreachFunc)).register()
598   }

I would like to print whole the RDD, I have seen I should use the collect() function instead of take(), I have done but I have gotten a compilation error.
does anyone help me?

BR

Em 3/25/14, 13:48, Sourav Chandra escreveu:
You can extend DStream and override print() method. Then you can create your own DSTream extending from this.


On Tue, Mar 25, 2014 at 6:07 PM, Eduardo Costa Alfaia <[hidden email]> wrote:
Hi Guys,
I think that I already did this question, but I don't remember if anyone has answered me. I would like changing in the function print() the quantity of words and the frequency number that are sent to driver's screen. The default value is 10.

Anyone could help me with this?

Best Regards

--
Informativa sulla Privacy: http://www.unibs.it/node/8155



--

Sourav Chandra

Senior Software Engineer

· · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · ·

[hidden email]

o: +91 80 4121 8723

m: +91 988 699 3746

skype: sourav.chandra

Livestream

"Ajmera Summit", First Floor, #3/D, 68 Ward, 3rd Cross, 7th C Main, 3rd Block, Koramangala Industrial Area,

Bangalore 560034

www.livestream.com



Informativa sulla Privacy: http://www.unibs.it/node/8155
Reply | Threaded
Open this post in threaded view
|

Re: Change print() in JavaNetworkWordCount

Sourav Chandra
This works just fine without any compilation error:

def print() {
    def foreachFunc = (rdd: RDD[T], time: Time) => {
      
      val total = rdd.collect().toList
      println ("-------------------------------------------")
      println ("Time: " + time)
      println ("-------------------------------------------")
      
      total.foreach(println)
//      val first11 = rdd.take(11)
//      println ("-------------------------------------------")
//      println ("Time: " + time)
//      println ("-------------------------------------------")
//      first11.take(10).foreach(println)
//      if (first11.size > 10) println("...")
      println()
    }
    new ForEachDStream(this, context.sparkContext.clean(foreachFunc)).register()
  }


On Wed, Mar 26, 2014 at 8:50 PM, Eduardo Costa Alfaia <[hidden email]> wrote:
Thanks Guys for reply, but I have found this piece of code in streaming:
 def print() {
585     def foreachFunc = (rdd: RDD[T], time: Time) => {
586       //val first11 = rdd.take(11)
587       //val first100 = rdd.take(100)
588         val total = rdd.collect()
589       println ("-------------------------------------------")
590       println ("Time: " + time)
591       println ("-------------------------------------------")
592       //first100.take(100).foreach(println)
593         total.collect().foreach(println)
594       //if (first100.size > 99) println("(...)")
595       println()
596     }
597     new ForEachDStream(this, context.sparkContext.clean(foreachFunc)).register()
598   }

I would like to print whole the RDD, I have seen I should use the collect() function instead of take(), I have done but I have gotten a compilation error.
does anyone help me?

BR

Em 3/25/14, 13:48, Sourav Chandra escreveu:
You can extend DStream and override print() method. Then you can create your own DSTream extending from this.


On Tue, Mar 25, 2014 at 6:07 PM, Eduardo Costa Alfaia <[hidden email]> wrote:
Hi Guys,
I think that I already did this question, but I don't remember if anyone has answered me. I would like changing in the function print() the quantity of words and the frequency number that are sent to driver's screen. The default value is 10.

Anyone could help me with this?

Best Regards

--
Informativa sulla Privacy: http://www.unibs.it/node/8155



--

Sourav Chandra

Senior Software Engineer

· · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · ·

[hidden email]

o: +91 80 4121 8723

m: +91 988 699 3746

skype: sourav.chandra

Livestream

"Ajmera Summit", First Floor, #3/D, 68 Ward, 3rd Cross, 7th C Main, 3rd Block, Koramangala Industrial Area,

Bangalore 560034

www.livestream.com



Informativa sulla Privacy: http://www.unibs.it/node/8155



--

Sourav Chandra

Senior Software Engineer

· · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · ·

[hidden email]

o: +91 80 4121 8723

m: +91 988 699 3746

skype: sourav.chandra

Livestream

"Ajmera Summit", First Floor, #3/D, 68 Ward, 3rd Cross, 7th C Main, 3rd Block, Koramangala Industrial Area,

Bangalore 560034

www.livestream.com

Reply | Threaded
Open this post in threaded view
|

Re: Change print() in JavaNetworkWordCount

eduardocalfaia
Thank you very much  Sourav

BR

Em 3/26/14, 17:29, Sourav Chandra escreveu:

> def print() {
>     def foreachFunc = (rdd: RDD[T], time: Time) => {
>       val total = rdd.collect().toList
>       println ("-------------------------------------------")
>       println ("Time: " + time)
>       println ("-------------------------------------------")
>       total.foreach(println)
> //      val first11 = rdd.take(11)
> //      println ("-------------------------------------------")
> //      println ("Time: " + time)
> //      println ("-------------------------------------------")
> //      first11.take(10).foreach(println)
> //      if (first11.size > 10) println("...")
>       println()
>     }
>     new ForEachDStream(this,
> context.sparkContext.clean(foreachFunc)).register()
>   }


--
Informativa sulla Privacy: http://www.unibs.it/node/8155