State of spark on scala 2.10

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

State of spark on scala 2.10

Aureliano Buendia
Hi,

I was going to give https://github.com/scala/pickling a try on spark to see how it would compare with kryo. Unfortunately, it only works with scala 2.10.3.

- Is there a time line for spark to work with scala 2.10?

- Is the 2.10 branch as stable as 2.9?

- What's blocking spark to work with 2.10?
Reply | Threaded
Open this post in threaded view
|

Re: State of spark on scala 2.10

Aaron Davidson
Scala 2.10.3 support was recently merged into master (#259). The branch is probably not as stable as 0.8.1, but things "should" work.

The 2.10 branch should be deleted, the only issue is there are some outstanding PRs against that branch that haven't been moved to master.


On Sat, Jan 4, 2014 at 7:11 PM, Aureliano Buendia <[hidden email]> wrote:
Hi,

I was going to give https://github.com/scala/pickling a try on spark to see how it would compare with kryo. Unfortunately, it only works with scala 2.10.3.

- Is there a time line for spark to work with scala 2.10?

- Is the 2.10 branch as stable as 2.9?

- What's blocking spark to work with 2.10?

Reply | Threaded
Open this post in threaded view
|

Re: State of spark on scala 2.10

Tathagata Das
Thumbs up on your plans of trying pickling. I was at the Scala Days 2013 presentation of pickling and it was very cool. Would be fun to see how does compile-time generation of serialization code perform against Kryo. 

TD


On Sat, Jan 4, 2014 at 7:56 PM, Aaron Davidson <[hidden email]> wrote:
Scala 2.10.3 support was recently merged into master (#259). The branch is probably not as stable as 0.8.1, but things "should" work.

The 2.10 branch should be deleted, the only issue is there are some outstanding PRs against that branch that haven't been moved to master.


On Sat, Jan 4, 2014 at 7:11 PM, Aureliano Buendia <[hidden email]> wrote:
Hi,

I was going to give https://github.com/scala/pickling a try on spark to see how it would compare with kryo. Unfortunately, it only works with scala 2.10.3.

- Is there a time line for spark to work with scala 2.10?

- Is the 2.10 branch as stable as 2.9?

- What's blocking spark to work with 2.10?


Reply | Threaded
Open this post in threaded view
|

Re: State of spark on scala 2.10

Aureliano Buendia
In reply to this post by Aaron Davidson
Good to know the next release will be on scala 2.10.

Meanwhile, does using the master branch mean that I will have to build my own AMI when launching an ec2 cluster? Also, is there a nightly binary build maven repo available for spark?


On Sun, Jan 5, 2014 at 3:56 AM, Aaron Davidson <[hidden email]> wrote:
Scala 2.10.3 support was recently merged into master (#259). The branch is probably not as stable as 0.8.1, but things "should" work.

The 2.10 branch should be deleted, the only issue is there are some outstanding PRs against that branch that haven't been moved to master.


On Sat, Jan 4, 2014 at 7:11 PM, Aureliano Buendia <[hidden email]> wrote:
Hi,

I was going to give https://github.com/scala/pickling a try on spark to see how it would compare with kryo. Unfortunately, it only works with scala 2.10.3.

- Is there a time line for spark to work with scala 2.10?

- Is the 2.10 branch as stable as 2.9?

- What's blocking spark to work with 2.10?


Reply | Threaded
Open this post in threaded view
|

Re: State of spark on scala 2.10

Patrick Wendell
You'll have to build your own. Also there are some packaging
differences in master (some bin/ scripts moved to sbin/) just to give
you a heads up.

On Sat, Jan 4, 2014 at 8:14 PM, Aureliano Buendia <[hidden email]> wrote:

> Good to know the next release will be on scala 2.10.
>
> Meanwhile, does using the master branch mean that I will have to build my
> own AMI when launching an ec2 cluster? Also, is there a nightly binary build
> maven repo available for spark?
>
>
> On Sun, Jan 5, 2014 at 3:56 AM, Aaron Davidson <[hidden email]> wrote:
>>
>> Scala 2.10.3 support was recently merged into master (#259). The branch is
>> probably not as stable as 0.8.1, but things "should" work.
>>
>> The 2.10 branch should be deleted, the only issue is there are some
>> outstanding PRs against that branch that haven't been moved to master.
>>
>>
>> On Sat, Jan 4, 2014 at 7:11 PM, Aureliano Buendia <[hidden email]>
>> wrote:
>>>
>>> Hi,
>>>
>>> I was going to give https://github.com/scala/pickling a try on spark to
>>> see how it would compare with kryo. Unfortunately, it only works with scala
>>> 2.10.3.
>>>
>>> - Is there a time line for spark to work with scala 2.10?
>>>
>>> - Is the 2.10 branch as stable as 2.9?
>>>
>>> - What's blocking spark to work with 2.10?
>>
>>
>
Reply | Threaded
Open this post in threaded view
|

Re: State of spark on scala 2.10

Patrick Wendell
I meant you'll need to build your own version of Spark. Typically we
do this by launching an existing AMI and then just building a new
version of spark and copying it to the slaves....

- Patrick

On Sat, Jan 4, 2014 at 8:44 PM, Patrick Wendell <[hidden email]> wrote:

> You'll have to build your own. Also there are some packaging
> differences in master (some bin/ scripts moved to sbin/) just to give
> you a heads up.
>
> On Sat, Jan 4, 2014 at 8:14 PM, Aureliano Buendia <[hidden email]> wrote:
>> Good to know the next release will be on scala 2.10.
>>
>> Meanwhile, does using the master branch mean that I will have to build my
>> own AMI when launching an ec2 cluster? Also, is there a nightly binary build
>> maven repo available for spark?
>>
>>
>> On Sun, Jan 5, 2014 at 3:56 AM, Aaron Davidson <[hidden email]> wrote:
>>>
>>> Scala 2.10.3 support was recently merged into master (#259). The branch is
>>> probably not as stable as 0.8.1, but things "should" work.
>>>
>>> The 2.10 branch should be deleted, the only issue is there are some
>>> outstanding PRs against that branch that haven't been moved to master.
>>>
>>>
>>> On Sat, Jan 4, 2014 at 7:11 PM, Aureliano Buendia <[hidden email]>
>>> wrote:
>>>>
>>>> Hi,
>>>>
>>>> I was going to give https://github.com/scala/pickling a try on spark to
>>>> see how it would compare with kryo. Unfortunately, it only works with scala
>>>> 2.10.3.
>>>>
>>>> - Is there a time line for spark to work with scala 2.10?
>>>>
>>>> - Is the 2.10 branch as stable as 2.9?
>>>>
>>>> - What's blocking spark to work with 2.10?
>>>
>>>
>>
Reply | Threaded
Open this post in threaded view
|

Re: State of spark on scala 2.10

Aureliano Buendia
So I should just lanuch an AMI from one of https://github.com/mesos/spark-ec2/tree/v2/ami-list and build the development version on it? Is it just a simple matter of copying the target to a directory, or does it need a lot of configurations?


On Sun, Jan 5, 2014 at 5:03 AM, Patrick Wendell <[hidden email]> wrote:
I meant you'll need to build your own version of Spark. Typically we
do this by launching an existing AMI and then just building a new
version of spark and copying it to the slaves....

- Patrick

On Sat, Jan 4, 2014 at 8:44 PM, Patrick Wendell <[hidden email]> wrote:
> You'll have to build your own. Also there are some packaging
> differences in master (some bin/ scripts moved to sbin/) just to give
> you a heads up.
>
> On Sat, Jan 4, 2014 at 8:14 PM, Aureliano Buendia <[hidden email]> wrote:
>> Good to know the next release will be on scala 2.10.
>>
>> Meanwhile, does using the master branch mean that I will have to build my
>> own AMI when launching an ec2 cluster? Also, is there a nightly binary build
>> maven repo available for spark?
>>
>>
>> On Sun, Jan 5, 2014 at 3:56 AM, Aaron Davidson <[hidden email]> wrote:
>>>
>>> Scala 2.10.3 support was recently merged into master (#259). The branch is
>>> probably not as stable as 0.8.1, but things "should" work.
>>>
>>> The 2.10 branch should be deleted, the only issue is there are some
>>> outstanding PRs against that branch that haven't been moved to master.
>>>
>>>
>>> On Sat, Jan 4, 2014 at 7:11 PM, Aureliano Buendia <[hidden email]>
>>> wrote:
>>>>
>>>> Hi,
>>>>
>>>> I was going to give https://github.com/scala/pickling a try on spark to
>>>> see how it would compare with kryo. Unfortunately, it only works with scala
>>>> 2.10.3.
>>>>
>>>> - Is there a time line for spark to work with scala 2.10?
>>>>
>>>> - Is the 2.10 branch as stable as 2.9?
>>>>
>>>> - What's blocking spark to work with 2.10?
>>>
>>>
>>

Reply | Threaded
Open this post in threaded view
|

Re: State of spark on scala 2.10

Patrick Wendell
I usually just use the existing launch scripts to create a correctly
sized cluster then just:

1. Copy spark/conf/* to /tmp
2. rm -rf spark/
3. Checkout and build new spark from github (you'll need to rename
incubator-spark to spark)
4. Copy conf files back
5. /root/spark-ec2/copy-dir --delete /root/spark


- Patrick

On Sat, Jan 4, 2014 at 9:14 PM, Aureliano Buendia <[hidden email]> wrote:

> So I should just lanuch an AMI from one of
> https://github.com/mesos/spark-ec2/tree/v2/ami-list and build the
> development version on it? Is it just a simple matter of copying the target
> to a directory, or does it need a lot of configurations?
>
>
>
> On Sun, Jan 5, 2014 at 5:03 AM, Patrick Wendell <[hidden email]> wrote:
>>
>> I meant you'll need to build your own version of Spark. Typically we
>> do this by launching an existing AMI and then just building a new
>> version of spark and copying it to the slaves....
>>
>> - Patrick
>>
>> On Sat, Jan 4, 2014 at 8:44 PM, Patrick Wendell <[hidden email]>
>> wrote:
>> > You'll have to build your own. Also there are some packaging
>> > differences in master (some bin/ scripts moved to sbin/) just to give
>> > you a heads up.
>> >
>> > On Sat, Jan 4, 2014 at 8:14 PM, Aureliano Buendia <[hidden email]>
>> > wrote:
>> >> Good to know the next release will be on scala 2.10.
>> >>
>> >> Meanwhile, does using the master branch mean that I will have to build
>> >> my
>> >> own AMI when launching an ec2 cluster? Also, is there a nightly binary
>> >> build
>> >> maven repo available for spark?
>> >>
>> >>
>> >> On Sun, Jan 5, 2014 at 3:56 AM, Aaron Davidson <[hidden email]>
>> >> wrote:
>> >>>
>> >>> Scala 2.10.3 support was recently merged into master (#259). The
>> >>> branch is
>> >>> probably not as stable as 0.8.1, but things "should" work.
>> >>>
>> >>> The 2.10 branch should be deleted, the only issue is there are some
>> >>> outstanding PRs against that branch that haven't been moved to master.
>> >>>
>> >>>
>> >>> On Sat, Jan 4, 2014 at 7:11 PM, Aureliano Buendia
>> >>> <[hidden email]>
>> >>> wrote:
>> >>>>
>> >>>> Hi,
>> >>>>
>> >>>> I was going to give https://github.com/scala/pickling a try on spark
>> >>>> to
>> >>>> see how it would compare with kryo. Unfortunately, it only works with
>> >>>> scala
>> >>>> 2.10.3.
>> >>>>
>> >>>> - Is there a time line for spark to work with scala 2.10?
>> >>>>
>> >>>> - Is the 2.10 branch as stable as 2.9?
>> >>>>
>> >>>> - What's blocking spark to work with 2.10?
>> >>>
>> >>>
>> >>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: State of spark on scala 2.10

Aureliano Buendia



On Sun, Jan 5, 2014 at 5:42 AM, Patrick Wendell <[hidden email]> wrote:
I usually just use the existing launch scripts to create a correctly
sized cluster then just:

1. Copy spark/conf/* to /tmp
2. rm -rf spark/
3. Checkout and build new spark from github (you'll need to rename
incubator-spark to spark)

I called 'sbt assembly' at this point, and it took ages to build the examples. What would be the minimal build command, just to build enough stuff for production?

 
4. Copy conf files back
5. /root/spark-ec2/copy-dir --delete /root/spark


- Patrick

On Sat, Jan 4, 2014 at 9:14 PM, Aureliano Buendia <[hidden email]> wrote:
> So I should just lanuch an AMI from one of
> https://github.com/mesos/spark-ec2/tree/v2/ami-list and build the
> development version on it? Is it just a simple matter of copying the target
> to a directory, or does it need a lot of configurations?
>
>
>
> On Sun, Jan 5, 2014 at 5:03 AM, Patrick Wendell <[hidden email]> wrote:
>>
>> I meant you'll need to build your own version of Spark. Typically we
>> do this by launching an existing AMI and then just building a new
>> version of spark and copying it to the slaves....
>>
>> - Patrick
>>
>> On Sat, Jan 4, 2014 at 8:44 PM, Patrick Wendell <[hidden email]>
>> wrote:
>> > You'll have to build your own. Also there are some packaging
>> > differences in master (some bin/ scripts moved to sbin/) just to give
>> > you a heads up.
>> >
>> > On Sat, Jan 4, 2014 at 8:14 PM, Aureliano Buendia <[hidden email]>
>> > wrote:
>> >> Good to know the next release will be on scala 2.10.
>> >>
>> >> Meanwhile, does using the master branch mean that I will have to build
>> >> my
>> >> own AMI when launching an ec2 cluster? Also, is there a nightly binary
>> >> build
>> >> maven repo available for spark?
>> >>
>> >>
>> >> On Sun, Jan 5, 2014 at 3:56 AM, Aaron Davidson <[hidden email]>
>> >> wrote:
>> >>>
>> >>> Scala 2.10.3 support was recently merged into master (#259). The
>> >>> branch is
>> >>> probably not as stable as 0.8.1, but things "should" work.
>> >>>
>> >>> The 2.10 branch should be deleted, the only issue is there are some
>> >>> outstanding PRs against that branch that haven't been moved to master.
>> >>>
>> >>>
>> >>> On Sat, Jan 4, 2014 at 7:11 PM, Aureliano Buendia
>> >>> <[hidden email]>
>> >>> wrote:
>> >>>>
>> >>>> Hi,
>> >>>>
>> >>>> I was going to give https://github.com/scala/pickling a try on spark
>> >>>> to
>> >>>> see how it would compare with kryo. Unfortunately, it only works with
>> >>>> scala
>> >>>> 2.10.3.
>> >>>>
>> >>>> - Is there a time line for spark to work with scala 2.10?
>> >>>>
>> >>>> - Is the 2.10 branch as stable as 2.9?
>> >>>>
>> >>>> - What's blocking spark to work with 2.10?
>> >>>
>> >>>
>> >>
>
>

Reply | Threaded
Open this post in threaded view
|

Re: State of spark on scala 2.10

Prashant Sharma
sbt assembly/assembly  

or if you are devloping spark i.e. working on spark code base itself not rolling a distribution. 
sbt assemble-deps

also 

there is a convenience script called make-distribution.sh 

HTH


On Sun, Jan 5, 2014 at 7:52 PM, Aureliano Buendia <[hidden email]> wrote:



On Sun, Jan 5, 2014 at 5:42 AM, Patrick Wendell <[hidden email]> wrote:
I usually just use the existing launch scripts to create a correctly
sized cluster then just:

1. Copy spark/conf/* to /tmp
2. rm -rf spark/
3. Checkout and build new spark from github (you'll need to rename
incubator-spark to spark)

I called 'sbt assembly' at this point, and it took ages to build the examples. What would be the minimal build command, just to build enough stuff for production?

 
4. Copy conf files back
5. /root/spark-ec2/copy-dir --delete /root/spark


- Patrick

On Sat, Jan 4, 2014 at 9:14 PM, Aureliano Buendia <[hidden email]> wrote:
> So I should just lanuch an AMI from one of
> https://github.com/mesos/spark-ec2/tree/v2/ami-list and build the
> development version on it? Is it just a simple matter of copying the target
> to a directory, or does it need a lot of configurations?
>
>
>
> On Sun, Jan 5, 2014 at 5:03 AM, Patrick Wendell <[hidden email]> wrote:
>>
>> I meant you'll need to build your own version of Spark. Typically we
>> do this by launching an existing AMI and then just building a new
>> version of spark and copying it to the slaves....
>>
>> - Patrick
>>
>> On Sat, Jan 4, 2014 at 8:44 PM, Patrick Wendell <[hidden email]>
>> wrote:
>> > You'll have to build your own. Also there are some packaging
>> > differences in master (some bin/ scripts moved to sbin/) just to give
>> > you a heads up.
>> >
>> > On Sat, Jan 4, 2014 at 8:14 PM, Aureliano Buendia <[hidden email]>
>> > wrote:
>> >> Good to know the next release will be on scala 2.10.
>> >>
>> >> Meanwhile, does using the master branch mean that I will have to build
>> >> my
>> >> own AMI when launching an ec2 cluster? Also, is there a nightly binary
>> >> build
>> >> maven repo available for spark?
>> >>
>> >>
>> >> On Sun, Jan 5, 2014 at 3:56 AM, Aaron Davidson <[hidden email]>
>> >> wrote:
>> >>>
>> >>> Scala 2.10.3 support was recently merged into master (#259). The
>> >>> branch is
>> >>> probably not as stable as 0.8.1, but things "should" work.
>> >>>
>> >>> The 2.10 branch should be deleted, the only issue is there are some
>> >>> outstanding PRs against that branch that haven't been moved to master.
>> >>>
>> >>>
>> >>> On Sat, Jan 4, 2014 at 7:11 PM, Aureliano Buendia
>> >>> <[hidden email]>
>> >>> wrote:
>> >>>>
>> >>>> Hi,
>> >>>>
>> >>>> I was going to give https://github.com/scala/pickling a try on spark
>> >>>> to
>> >>>> see how it would compare with kryo. Unfortunately, it only works with
>> >>>> scala
>> >>>> 2.10.3.
>> >>>>
>> >>>> - Is there a time line for spark to work with scala 2.10?
>> >>>>
>> >>>> - Is the 2.10 branch as stable as 2.9?
>> >>>>
>> >>>> - What's blocking spark to work with 2.10?
>> >>>
>> >>>
>> >>
>
>




--
Prashant