Anyone interested in Remote Shuffle Service

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Anyone interested in Remote Shuffle Service

bo yang
Hi Spark Users,

Uber open sourced Remote Shuffle Service ( https://github.com/uber/RemoteShuffleService ) recently. It works with open source Spark version without code change needed, and could store shuffle data on separate machines other than Spark executors.

Anyone interested to try? Also we are still actively developing. Please let us know if you are interested to work together on it as well :)

Best,
Bo