Help with groupByKey

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Help with groupByKey

David Thomas
I have an RDD of (K, Array[V]) pairs.

For example: ((key1, (1,2,3)), (key2, (3,2,4)), (key1, (4,3,2)))

How can I do a groupByKey such that I get back an RDD of the form (K, Array[V]) pairs.

Ex: ((key1, (1,2,3,4,3,2)), (key2, (3,2,4)))
Reply | Threaded
Open this post in threaded view
|

Re: Help with groupByKey

Andrew Ash

rdd.reduceByKey(_+_) using list concatenation?

Sent from my mobile phone

On Mar 2, 2014 7:05 PM, "David Thomas" <[hidden email]> wrote:
I have an RDD of (K, Array[V]) pairs.

For example: ((key1, (1,2,3)), (key2, (3,2,4)), (key1, (4,3,2)))

How can I do a groupByKey such that I get back an RDD of the form (K, Array[V]) pairs.

Ex: ((key1, (1,2,3,4,3,2)), (key2, (3,2,4)))
Reply | Threaded
Open this post in threaded view
|

Re: Help with groupByKey

Cheng Lian-3
Actually it should be rdd.reduceByKey(_ ++ _)

On Mar 3, 2014, at 11:56, Andrew Ash <[hidden email]> wrote:

rdd.reduceByKey(_+_) using list concatenation?

Sent from my mobile phone

On Mar 2, 2014 7:05 PM, "David Thomas" <[hidden email]> wrote:
I have an RDD of (K, Array[V]) pairs.

For example: ((key1, (1,2,3)), (key2, (3,2,4)), (key1, (4,3,2)))

How can I do a groupByKey such that I get back an RDD of the form (K, Array[V]) pairs.

Ex: ((key1, (1,2,3,4,3,2)), (key2, (3,2,4)))

Reply | Threaded
Open this post in threaded view
|

Re: Help with groupByKey

Cheng Lian-3
In reply to this post by Andrew Ash
Actually it should be rdd.reduceByKey(_ ++ _)

On Mar 3, 2014, at 11:56, Andrew Ash <[hidden email]> wrote:

rdd.reduceByKey(_+_) using list concatenation?

Sent from my mobile phone

On Mar 2, 2014 7:05 PM, "David Thomas" <[hidden email]> wrote:
I have an RDD of (K, Array[V]) pairs.

For example: ((key1, (1,2,3)), (key2, (3,2,4)), (key1, (4,3,2)))

How can I do a groupByKey such that I get back an RDD of the form (K, Array[V]) pairs.

Ex: ((key1, (1,2,3,4,3,2)), (key2, (3,2,4)))