Pass config file through spark-submit

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Pass config file through spark-submit

James Starks
I have a config file that exploits type safe config library located on the local file system, and want to submit that file through spark-submit so that spark program can read customized parameters. For instance,

my.app {
  db {
    host = domain.cc
    port = 1234
    db = dbname
    user = myuser
    passwd = mypass
  }
}

Spark submit code looks like

    spark-submit --class "my.app.Sample" --master local[*] --conf "spark.executor.extraJavaOptions=-Dconfig.file=/path/to/conf/myapp.conf" /path/to/my-app.jar

But the program can not read the parameters such as db, user, host, and so on in my conf file.

Passing with --files /path/to/myapp.conf doesn't work either.

What is the correct way to submit that kind of conf file so that my spark job can read customized parameters from there?

Thanks



Reply | Threaded
Open this post in threaded view
|

Re: Pass config file through spark-submit

yujhe.li
So can you read the file on executor side?
I think the file passed by --files my.app.conf would be added under
classpath, and you can use it directly.



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Pass config file through spark-submit

James Starks
Accidentally to get it working, though don't thoroughly understand why (So far as I know, it's to configure in allowing executor refers to the conf file after copying to executors' working dir). Basically it's a combination of parameters --conf, --files, and --driver-class-path, instead of any single parameter.

spark-submit --class pkg.to.MyApp --master local[*] --conf "spark.executor.extraClassPath=-Dconfig.file=<myfile.conf>" --files <conf/myfile.conf> --driver-class-path "</absolute/path/to/conf/dir>"

--conf requires to pass the conf file name e.g. myfile.conf along with spark executor class path as directive.

--files passes the conf file associated from the context root e.g. executing under dir <my-project>, under which it contains folders such as conf, logs, work and so on. The conf file i.e. myfile.conf is located under conf folder.

--driver-class-path points to the conf directory with absolute path.


‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On August 17, 2018 3:00 AM, yujhe.li <[hidden email]> wrote:

> So can you read the file on executor side?
> I think the file passed by --files my.app.conf would be added under
> classpath, and you can use it directly.
>
>
> --------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
>
> ----------------------------------------------------------------
>
> To unsubscribe e-mail: [hidden email]



---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]