Spark 0.8.1 on Amazon Elastic MapReduce

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Spark 0.8.1 on Amazon Elastic MapReduce

parvizd
Spark community,

Wanted to let you know that the version of Spark and Shark on Amazon Elastic MapReduce has been updated to 0.8.1. This new version provides a much better experience in terms of stability and performance but also supports the following features: 
  • Integration with Amazon Cloudwatch
  • Integration of Spark Streaming with Amazon Kinesis. 
  • Automatic log shipping to S3
For a complete detail of the features Spark on EMR provides, please see the following article: http://aws.amazon.com/articles/4926593393724923

And yes I'm working hard to push another update to support 0.9.0 :)

What would be great is to hear from the community on what other features you like to see on Spark on EMR. For example, how useful is autoscaling for Spark? Any other features you like to see?

Thanks,

Parviz Deyhim

Solutions Architect

Amazon Web Services

E: [hidden email]

M:  408.315.2305

 

Description: Description: Description: C:\Users\aiden\AppData\Local\Microsoft\Windows\Temporary Internet Files\Content.Word\aws.gif

Reply | Threaded
Open this post in threaded view
|

Re: Spark 0.8.1 on Amazon Elastic MapReduce

MLnick
Thanks Parviz, this looks great and good to see it getting updated. Look forward to 0.9.0!

A perhaps stupid question - where does the KinesisWordCount example live? Is that an Amazon example, since I don't see it under the streaming examples included in the Spark project. If it's a third party example is it possible to get the code?

Thanks
Nick


On Fri, Feb 14, 2014 at 6:53 PM, Deyhim, Parviz <[hidden email]> wrote:
Spark community,

Wanted to let you know that the version of Spark and Shark on Amazon Elastic MapReduce has been updated to 0.8.1. This new version provides a much better experience in terms of stability and performance but also supports the following features: 
  • Integration with Amazon Cloudwatch
  • Integration of Spark Streaming with Amazon Kinesis. 
  • Automatic log shipping to S3
For a complete detail of the features Spark on EMR provides, please see the following article: http://aws.amazon.com/articles/4926593393724923

And yes I'm working hard to push another update to support 0.9.0 :)

What would be great is to hear from the community on what other features you like to see on Spark on EMR. For example, how useful is autoscaling for Spark? Any other features you like to see?

Thanks,

Parviz Deyhim

Solutions Architect

Amazon Web Services

E: [hidden email]

M:  <a href="tel:408.315.2305" value="+14083152305" target="_blank">408.315.2305

 

Description: Description: Description: C:\Users\aiden\AppData\Local\Microsoft\Windows\Temporary Internet Files\Content.Word\aws.gif


Reply | Threaded
Open this post in threaded view
|

Re: Spark 0.8.1 on Amazon Elastic MapReduce

Tathagata Das
That's interesting! I am curious to find out as well. 

TD


On Fri, Feb 14, 2014 at 9:28 AM, Nick Pentreath <[hidden email]> wrote:
Thanks Parviz, this looks great and good to see it getting updated. Look forward to 0.9.0!

A perhaps stupid question - where does the KinesisWordCount example live? Is that an Amazon example, since I don't see it under the streaming examples included in the Spark project. If it's a third party example is it possible to get the code?

Thanks
Nick


On Fri, Feb 14, 2014 at 6:53 PM, Deyhim, Parviz <[hidden email]> wrote:
Spark community,

Wanted to let you know that the version of Spark and Shark on Amazon Elastic MapReduce has been updated to 0.8.1. This new version provides a much better experience in terms of stability and performance but also supports the following features: 
  • Integration with Amazon Cloudwatch
  • Integration of Spark Streaming with Amazon Kinesis. 
  • Automatic log shipping to S3
For a complete detail of the features Spark on EMR provides, please see the following article: http://aws.amazon.com/articles/4926593393724923

And yes I'm working hard to push another update to support 0.9.0 :)

What would be great is to hear from the community on what other features you like to see on Spark on EMR. For example, how useful is autoscaling for Spark? Any other features you like to see?

Thanks,

Parviz Deyhim

Solutions Architect

Amazon Web Services

E: [hidden email]

M:  <a href="tel:408.315.2305" value="+14083152305" target="_blank">408.315.2305

 

Description: Description: Description: C:\Users\aiden\AppData\Local\Microsoft\Windows\Temporary Internet Files\Content.Word\aws.gif