interleave partitions?

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

interleave partitions?

Walrus theCat
Hi,

I want to do something like this:

rdd3 = rdd1.coalesce(N).partitions.zip(rdd2.coalesce(N).partitions)

I realize the above will get me something like Array[(partition,partition)].

I hope you see what I'm going for here -- any tips on how to accomplish this?

Thanks
Reply | Threaded
Open this post in threaded view
|

Re: interleave partitions?

Walrus theCat
Answering my own question here.  This may not be efficient, but this is what I came up with:

rdd1.coalesce(N).glom.zip(rdd2.coalesce(N).glom).map { case(x,y) => x++y}


On Wed, Mar 26, 2014 at 11:11 AM, Walrus theCat <[hidden email]> wrote:
Hi,

I want to do something like this:

rdd3 = rdd1.coalesce(N).partitions.zip(rdd2.coalesce(N).partitions)

I realize the above will get me something like Array[(partition,partition)].

I hope you see what I'm going for here -- any tips on how to accomplish this?

Thanks