can I model any arbitrary data structure as an RDD?

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

can I model any arbitrary data structure as an RDD?

kant kodali
Hi All,

I am wondering if I can model any arbitrary data structure as an RDD? For example, can I model, Red-black trees, Suffix Trees, Radix Trees, Splay Trees, Fibonacci heaps, Tries, Linked Lists etc as RDD's? If so, how?

To implement a custom RDD I have to implement compute and getPartitions functions so does this mean that as long as I can store the above data structures into some storage and implement the compute and getPatitions functions am I good? I wonder if every data structure is parallelizable in the first place?

Thanks!