Client APIs for Accessing Spark Data Frames Directly

Previous Topic Next Topic
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

Client APIs for Accessing Spark Data Frames Directly

Artemis User

Is there anyway to access the Data Frames content directly/interactively via some client access APIs?  Some background info:

  1. We have a Java client application that uses spark launcher to submit a spark job to a spark master.
  2. The default spark launcher API has only a handle API that provides job status information
  3. It is possible to use the Hive JDBC to access data via the built-in thrifty server, but the data have to be saved into hive tables, not sure if these tables have to be persisted on disk and require huge overhead for data I/O.  Since DF is in memory, it would be more convenient and faster to access the data directly in memory if such APIs are available.

Thanks!  -- ND