[Spark R] Proposal: Exposing RBackend in RRunner

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

[Spark R] Proposal: Exposing RBackend in RRunner

Spark Users,

In SparkR, RBackend is created in RRunner.main(). This in particular makes it difficult to control or use the RBackend. For my use case, I am looking to access the JVMObjectTracker that RBackend maintains for SparkR dataframes.

Analogously, pyspark starts a py4j.GatewayServer in PythonRunner.main(). It's then possible to start a ClientServer that then has access to the object bindings between Python/Java. 

Is there something similar for SparkR? Or a reasonable way to expose RBackend?


Jeremy Liu