How to submit a job via REST API?

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

How to submit a job via REST API?

Zhou Yang
Dear experts,

I found a convenient way to submit job via Rest API at https://gist.github.com/arturmkrtchyan/5d8559b2911ac951d34a#file-submit_job-sh.
But I did not know whether can I append `—conf` parameter like what I did in spark-submit. Can someone can help me with this issue?

Regards, Yang

Reply | Threaded
Open this post in threaded view
|

Re: How to submit a job via REST API?

Dennis Suhari
Hi Yang,

I am using Livy Server for submitting jobs.

Br,

Dennis



Von meinem iPhone gesendet

Am 24.11.2020 um 03:34 schrieb Zhou Yang <[hidden email]>:


Dear experts,

I found a convenient way to submit job via Rest API at https://gist.github.com/arturmkrtchyan/5d8559b2911ac951d34a#file-submit_job-sh.
But I did not know whether can I append `—conf` parameter like what I did in spark-submit. Can someone can help me with this issue?

Regards, Yang

Reply | Threaded
Open this post in threaded view
|

Re: How to submit a job via REST API?

Sonal Goyal
You should be able to supply the --conf and its values as part of appArgs argument

Cheers,
Sonal
Nube Technologies 
Join me at
Data Con LA Oct 23 | Big Data Conference Europe. Nov 24 | GIDS AI/ML Dec 3




On Tue, Nov 24, 2020 at 11:31 AM Dennis Suhari <[hidden email]> wrote:
Hi Yang,

I am using Livy Server for submitting jobs.

Br,

Dennis



Von meinem iPhone gesendet

Am 24.11.2020 um 03:34 schrieb Zhou Yang <[hidden email]>:


Dear experts,

I found a convenient way to submit job via Rest API at https://gist.github.com/arturmkrtchyan/5d8559b2911ac951d34a#file-submit_job-sh.
But I did not know whether can I append `—conf` parameter like what I did in spark-submit. Can someone can help me with this issue?

Regards, Yang

Reply | Threaded
Open this post in threaded view
|

Re: How to submit a job via REST API?

vaquar khan

On Wed, Nov 25, 2020 at 12:40 AM Sonal Goyal <[hidden email]> wrote:
You should be able to supply the --conf and its values as part of appArgs argument

Cheers,
Sonal
Nube Technologies 
Join me at
Data Con LA Oct 23 | Big Data Conference Europe. Nov 24 | GIDS AI/ML Dec 3




On Tue, Nov 24, 2020 at 11:31 AM Dennis Suhari <[hidden email]> wrote:
Hi Yang,

I am using Livy Server for submitting jobs.

Br,

Dennis



Von meinem iPhone gesendet

Am 24.11.2020 um 03:34 schrieb Zhou Yang <[hidden email]>:


Dear experts,

I found a convenient way to submit job via Rest API at https://gist.github.com/arturmkrtchyan/5d8559b2911ac951d34a#file-submit_job-sh.
But I did not know whether can I append `—conf` parameter like what I did in spark-submit. Can someone can help me with this issue?

Regards, Yang



--
Regards,
Vaquar Khan
+1 -224-436-0783
Greater Chicago
Reply | Threaded
Open this post in threaded view
|

Re: How to submit a job via REST API?

Zhou Yang
Hi all,

I found the solution through the source code. Appending the —conf k-v into `sparkProperties` work.
For example:

./spark-submit \
—conf foo=bar \
xxx

equals to

{
“xxx” : “yyy”,
sparkProperties” : {
“foo": "bar"
}
}

Thanks for your reply.

2020年11月25日 下午3:55,vaquar khan <[hidden email]> 写道:


On Wed, Nov 25, 2020 at 12:40 AM Sonal Goyal <[hidden email]> wrote:
You should be able to supply the --conf and its values as part of appArgs argument

Cheers,
Sonal
Nube Technologies 
Join me at
Data Con LA Oct 23 | Big Data Conference Europe. Nov 24 | GIDS AI/ML Dec 3




On Tue, Nov 24, 2020 at 11:31 AM Dennis Suhari <[hidden email]> wrote:
Hi Yang,

I am using Livy Server for submitting jobs.

Br,

Dennis



Von meinem iPhone gesendet

Am 24.11.2020 um 03:34 schrieb Zhou Yang <[hidden email]>:


Dear experts,

I found a convenient way to submit job via Rest API at https://gist.github.com/arturmkrtchyan/5d8559b2911ac951d34a#file-submit_job-sh.
But I did not know whether can I append `—conf` parameter like what I did in spark-submit. Can someone can help me with this issue?

Regards, Yang



--
Regards,
Vaquar Khan
+1 -224-436-0783
Greater Chicago

Reply | Threaded
Open this post in threaded view
|

Re: How to submit a job via REST API?

Daniel de Oliveira Mantovani
In my opinion this should be part of the official documentation. Amazing work Zhou Yang.

On Wed, Nov 25, 2020 at 5:45 AM Zhou Yang <[hidden email]> wrote:
Hi all,

I found the solution through the source code. Appending the —conf k-v into `sparkProperties` work.
For example:

./spark-submit \
—conf foo=bar \
xxx

equals to

{
“xxx” : “yyy”,
sparkProperties” : {
“foo": "bar"
}
}

Thanks for your reply.

2020年11月25日 下午3:55,vaquar khan <[hidden email]> 写道:


On Wed, Nov 25, 2020 at 12:40 AM Sonal Goyal <[hidden email]> wrote:
You should be able to supply the --conf and its values as part of appArgs argument

Cheers,
Sonal
Nube Technologies 
Join me at
Data Con LA Oct 23 | Big Data Conference Europe. Nov 24 | GIDS AI/ML Dec 3




On Tue, Nov 24, 2020 at 11:31 AM Dennis Suhari <[hidden email]> wrote:
Hi Yang,

I am using Livy Server for submitting jobs.

Br,

Dennis



Von meinem iPhone gesendet

Am 24.11.2020 um 03:34 schrieb Zhou Yang <[hidden email]>:


Dear experts,

I found a convenient way to submit job via Rest API at https://gist.github.com/arturmkrtchyan/5d8559b2911ac951d34a#file-submit_job-sh.
But I did not know whether can I append `—conf` parameter like what I did in spark-submit. Can someone can help me with this issue?

Regards, Yang



--
Regards,
Vaquar Khan
+1 -224-436-0783
Greater Chicago



--

--
Daniel Mantovani