Tachyon RDD Caching

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Tachyon RDD Caching

arosenberger
Hi, in this pull request Spark adds the ability to store RDDs in off heap memory via Tachyon: https://github.com/apache/incubator-spark/pull/468

I was wondering if the broadcast variable API would be opened up in a similar manner? From the source of e.g. HttpBroadcast, it is currently hard coded which would prevent Tachyon storage: SparkEnv.get.blockManager.putSingle(blockId, value_, StorageLevel.MEMORY_AND_DISK, false)

Thanks,
Adam