[Spark RDD] Persisting Spark RDDs across spark contexts/applications - options

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

[Spark RDD] Persisting Spark RDDs across spark contexts/applications - options

Boris Litvak

I would like to cache Apache Spark RDDs and share them between Spark applications.

Alluxio (Tachyon), Redis & Ignite all offer such capabilities.

For instance, see Ignite's proposal:

Are there any comparison studies on performance/maintenance burden/installation experience of the above frameworks?

If you have you had any experience using spark with any of these technologies, please share.

Thanks, Boris

 

Reply | Threaded
Open this post in threaded view
|

Re: [Spark RDD] Persisting Spark RDDs across spark contexts/applications - options

Bin Fan

On Thu, Jun 4, 2020 at 12:26 AM Boris Litvak <[hidden email]> wrote:

I would like to cache Apache Spark RDDs and share them between Spark applications.

Alluxio (Tachyon), Redis & Ignite all offer such capabilities.

For instance, see Ignite's proposal:

Are there any comparison studies on performance/maintenance burden/installation experience of the above frameworks?

If you have you had any experience using spark with any of these technologies, please share.

Thanks, Boris