Using P4J Plugins with Spark

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Using P4J Plugins with Spark

Shashanka Balakuntala
Hi users,
I'm a bit of newbie to spark infrastructure. And i have a small doubt.
I have a maven project with plugins generated separately in a folder and normal java command to run is as follows:
`java -Dp4j.pluginsDir=./plugins -jar /path/to/jar`

Now when I run this program in local with spark-submit with standalone cluster(not cluster mode) the program compiles and plugins are in "plugins" folder in the $SPARK_HOME and it is getting recognised.
The same is not the case in cluster mode. It says the Extenstion point is not loaded. Please advise on how can i create a folder which can be shared among the workers in "plugin" folder.

PS: HDFS is not an options as we dont have a different setup

Thanks.


Regards
  Shashanka Balakuntala Srinivasa
      
Reply | Threaded
Open this post in threaded view
|

Re: Using P4J Plugins with Spark

tnist
You may want to make sure you include the jar of P4J and your plugins as part of the following so that both the driver and executors have access.  If HDFS is out then you could 
make a common mount point on each of the executor nodes so they have access to the classes.

  • spark-submit --jars /common/path/to/jars
  • spark.driver.extraClassPath or it's alias --driver-class-path to set extra classpaths on the node running the driver.
  • spark.executor.extraClassPath to set extra class path on the Worker nodes.

On Tue, Apr 21, 2020 at 1:13 AM Shashanka Balakuntala <[hidden email]> wrote:
Hi users,
I'm a bit of newbie to spark infrastructure. And i have a small doubt.
I have a maven project with plugins generated separately in a folder and normal java command to run is as follows:
`java -Dp4j.pluginsDir=./plugins -jar /path/to/jar`

Now when I run this program in local with spark-submit with standalone cluster(not cluster mode) the program compiles and plugins are in "plugins" folder in the $SPARK_HOME and it is getting recognised.
The same is not the case in cluster mode. It says the Extenstion point is not loaded. Please advise on how can i create a folder which can be shared among the workers in "plugin" folder.

PS: HDFS is not an options as we dont have a different setup

Thanks.


Regards
  Shashanka Balakuntala Srinivasa