Replicating RDD elements

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Replicating RDD elements

David Thomas
How can we replicate RDD elements? Say I have 1 element and 100 nodes in the cluster. I need to replicate this one item on all the nodes i.e. effectively create an RDD of 100 elements.
Reply | Threaded
Open this post in threaded view
|

Re: Replicating RDD elements

Sonal Goyal
Hi David,

I am sorry but your question is not clear to me. Are you talking about taking some value and sharing it across your cluster so that it is present on all the nodes? You can look at Spark's broadcasting in that case. On the other hand, if you want to take one item and create an RDD of 100 or some other number of items, you could do a flatMap. Does that help?

Best Regards,
Sonal
Nube Technologies 






On Fri, Mar 28, 2014 at 9:24 AM, David Thomas <[hidden email]> wrote:
How can we replicate RDD elements? Say I have 1 element and 100 nodes in the cluster. I need to replicate this one item on all the nodes i.e. effectively create an RDD of 100 elements.

Reply | Threaded
Open this post in threaded view
|

Re: Replicating RDD elements

David Thomas
That helps! Thank you.


On Fri, Mar 28, 2014 at 12:36 AM, Sonal Goyal <[hidden email]> wrote:
Hi David,

I am sorry but your question is not clear to me. Are you talking about taking some value and sharing it across your cluster so that it is present on all the nodes? You can look at Spark's broadcasting in that case. On the other hand, if you want to take one item and create an RDD of 100 or some other number of items, you could do a flatMap. Does that help?

Best Regards,
Sonal
Nube Technologies 






On Fri, Mar 28, 2014 at 9:24 AM, David Thomas <[hidden email]> wrote:
How can we replicate RDD elements? Say I have 1 element and 100 nodes in the cluster. I need to replicate this one item on all the nodes i.e. effectively create an RDD of 100 elements.