Actors and sparkcontext actions

classic Classic list List threaded Threaded
5 messages Options
od
Reply | Threaded
Open this post in threaded view
|

Actors and sparkcontext actions

od
Can someone point me to a simple, short code example of creating a basic
Actor that gets a context and runs an operation such as .textFile.count?
I am trying to figure out how to create just a basic actor that gets a
message like this:

case class Msg(filename:String, ctx: SparkContext)

and then something like this:

class HelloActor extends Actor {
     import context.dispatcher

     def receive = {
         case Msg(fn,ctx) => {
             // get the count here!
             // cts.textFile(fn).count
         }
         case _ => println("huh?")
     }
}

Where I would want to do something like:

val conf = new
SparkConf().setMaster("spark://192.168.10.29:7077").setAppName("Hello").setSparkHome("/Users/maketo/plainvanilla/spark-0.9")
val sc = new SparkContext(conf)
val system = ActorSystem("mySystem")

val helloActor1 = system.actorOf( Props[ HelloActor], name = "helloactor1")
helloActor1 ! new Msg("test.json",sc)

Thanks,
Ognen
Reply | Threaded
Open this post in threaded view
|

RE: Actors and sparkcontext actions

Suraj Satishkumar Sheth
Hi Ognen,
See if this helps. I was working on this :

class MyClass[T](sc : SparkContext, flag1 : Boolean, rdd : RDD[T], hdfsPath : String) extends Actor {

  def act(){
    if(flag1) this.process()
    else this.count
  }
 
  private def process(){
    println(sc.textFile(hdfsPath).count)
    //do the processing
  }
 
  private def count(){
   println(rdd.count)
   //do the counting
  }

}

Thanks and Regards,
Suraj Sheth


-----Original Message-----
From: Ognen Duzlevski [mailto:[hidden email]]
Sent: 27 February 2014 01:09
To: [hidden email]
Subject: Actors and sparkcontext actions

Can someone point me to a simple, short code example of creating a basic Actor that gets a context and runs an operation such as .textFile.count?
I am trying to figure out how to create just a basic actor that gets a message like this:

case class Msg(filename:String, ctx: SparkContext)

and then something like this:

class HelloActor extends Actor {
     import context.dispatcher

     def receive = {
         case Msg(fn,ctx) => {
             // get the count here!
             // cts.textFile(fn).count
         }
         case _ => println("huh?")
     }
}

Where I would want to do something like:

val conf = new
SparkConf().setMaster("spark://192.168.10.29:7077").setAppName("Hello").setSparkHome("/Users/maketo/plainvanilla/spark-0.9")
val sc = new SparkContext(conf)
val system = ActorSystem("mySystem")

val helloActor1 = system.actorOf( Props[ HelloActor], name = "helloactor1")
helloActor1 ! new Msg("test.json",sc)

Thanks,
Ognen
Reply | Threaded
Open this post in threaded view
|

Re: Actors and sparkcontext actions

Ognen Duzlevski-2
Suraj, I posted to this list a link to my blog where I detail how to do
a simple actor/sparkcontext thing with the added obstacle of it being
within a Scalatra servlet.

Thanks for the code!
Ognen

On 3/4/14, 3:20 AM, Suraj Satishkumar Sheth wrote:

> Hi Ognen,
> See if this helps. I was working on this :
>
> class MyClass[T](sc : SparkContext, flag1 : Boolean, rdd : RDD[T], hdfsPath : String) extends Actor {
>
>    def act(){
>      if(flag1) this.process()
>      else this.count
>    }
>    
>    private def process(){
>      println(sc.textFile(hdfsPath).count)
>      //do the processing
>    }
>    
>    private def count(){
>     println(rdd.count)
>     //do the counting
>    }
>
> }
>
> Thanks and Regards,
> Suraj Sheth
>
>
> -----Original Message-----
> From: Ognen Duzlevski [mailto:[hidden email]]
> Sent: 27 February 2014 01:09
> To: [hidden email]
> Subject: Actors and sparkcontext actions
>
> Can someone point me to a simple, short code example of creating a basic Actor that gets a context and runs an operation such as .textFile.count?
> I am trying to figure out how to create just a basic actor that gets a message like this:
>
> case class Msg(filename:String, ctx: SparkContext)
>
> and then something like this:
>
> class HelloActor extends Actor {
>       import context.dispatcher
>
>       def receive = {
>           case Msg(fn,ctx) => {
>               // get the count here!
>               // cts.textFile(fn).count
>           }
>           case _ => println("huh?")
>       }
> }
>
> Where I would want to do something like:
>
> val conf = new
> SparkConf().setMaster("spark://192.168.10.29:7077").setAppName("Hello").setSparkHome("/Users/maketo/plainvanilla/spark-0.9")
> val sc = new SparkContext(conf)
> val system = ActorSystem("mySystem")
>
> val helloActor1 = system.actorOf( Props[ HelloActor], name = "helloactor1")
> helloActor1 ! new Msg("test.json",sc)
>
> Thanks,
> Ognen

--
Some people, when confronted with a problem, think "I know, I'll use regular expressions." Now they have two problems.
-- Jamie Zawinski

Reply | Threaded
Open this post in threaded view
|

Re: Actors and sparkcontext actions

Debasish Das
Hi Ognen,

Any particular reason of choosing scalatra over options like play or spray ? 

Is scalatra much better in serving apis or is it due to similarity with ruby's sinatra ?

Did you try the other options and then pick scalatra ?

Thanks.
Deb



On Tue, Mar 4, 2014 at 4:50 AM, Ognen Duzlevski <[hidden email]> wrote:
Suraj, I posted to this list a link to my blog where I detail how to do a simple actor/sparkcontext thing with the added obstacle of it being within a Scalatra servlet.

Thanks for the code!
Ognen


On 3/4/14, 3:20 AM, Suraj Satishkumar Sheth wrote:
Hi Ognen,
See if this helps. I was working on this :

class MyClass[T](sc : SparkContext, flag1 : Boolean, rdd : RDD[T], hdfsPath : String) extends Actor {

   def act(){
     if(flag1) this.process()
     else this.count
   }
      private def process(){
     println(sc.textFile(hdfsPath).count)
     //do the processing
   }
      private def count(){
    println(rdd.count)
    //do the counting
   }

}

Thanks and Regards,
Suraj Sheth


-----Original Message-----
From: Ognen Duzlevski [mailto:[hidden email]]
Sent: 27 February 2014 01:09
To: [hidden email]
Subject: Actors and sparkcontext actions

Can someone point me to a simple, short code example of creating a basic Actor that gets a context and runs an operation such as .textFile.count?
I am trying to figure out how to create just a basic actor that gets a message like this:

case class Msg(filename:String, ctx: SparkContext)

and then something like this:

class HelloActor extends Actor {
      import context.dispatcher

      def receive = {
          case Msg(fn,ctx) => {
              // get the count here!
              // cts.textFile(fn).count
          }
          case _ => println("huh?")
      }
}

Where I would want to do something like:

val conf = new
SparkConf().setMaster("spark://192.168.10.29:7077").setAppName("Hello").setSparkHome("/Users/maketo/plainvanilla/spark-0.9")
val sc = new SparkContext(conf)
val system = ActorSystem("mySystem")

val helloActor1 = system.actorOf( Props[ HelloActor], name = "helloactor1")
helloActor1 ! new Msg("test.json",sc)

Thanks,
Ognen

--
Some people, when confronted with a problem, think "I know, I'll use regular expressions." Now they have two problems.
-- Jamie Zawinski


Reply | Threaded
Open this post in threaded view
|

Re: Actors and sparkcontext actions

Ognen Duzlevski-2
Deb,

On 3/4/14, 9:02 AM, Debasish Das wrote:
> Hi Ognen,
>
> Any particular reason of choosing scalatra over options like play or
> spray ?
>
> Is scalatra much better in serving apis or is it due to similarity
> with ruby's sinatra ?
>
> Did you try the other options and then pick scalatra ?
Not really. I just happen to like Scalatra, it is easy with read and
easy to write. If it did not work out I would have gone probably with
something like Unfiltered.
Ognen