Spark - configuration setting doesn't work

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Spark - configuration setting doesn't work

Chetan Khatri
Hi Users,

I am setting spark configuration in below way;

val spark = SparkSession.builder().appName(APP_NAME).getOrCreate()

spark.conf.set("spark.speculation", "false")
spark.conf.set("spark.broadcast.compress", "true")
spark.conf.set("spark.sql.broadcastTimeout", "36000")
spark.conf.set("spark.network.timeout", "2500s")
spark.conf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
spark.conf.set("spark.driver.memory", "10g")
spark.conf.set("spark.executor.memory", "10g")

import spark.implicits._

and submitting spark job with spark - submit. but none of the above configuration is
getting reflected to the job, I have checked at Spark-UI.
I know setting up like this while creation of spark object, it's working well.

val spark = SparkSession.builder().appName(APP_NAME)
.config("spark.network.timeout", "1500s")
.config("spark.broadcast.compress", "true")
.config("spark.sql.broadcastTimeout", "36000")
.getOrCreate()

import spark.implicits._

Can someone please throw light?
Reply | Threaded
Open this post in threaded view
|

Re: Spark - configuration setting doesn't work

Chetan Khatri
Could someone please help me.

On Thu, Oct 17, 2019 at 7:29 PM Chetan Khatri <[hidden email]> wrote:
Hi Users,

I am setting spark configuration in below way;

val spark = SparkSession.builder().appName(APP_NAME).getOrCreate()

spark.conf.set("spark.speculation", "false")
spark.conf.set("spark.broadcast.compress", "true")
spark.conf.set("spark.sql.broadcastTimeout", "36000")
spark.conf.set("spark.network.timeout", "2500s")
spark.conf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
spark.conf.set("spark.driver.memory", "10g")
spark.conf.set("spark.executor.memory", "10g")

import spark.implicits._

and submitting spark job with spark - submit. but none of the above configuration is
getting reflected to the job, I have checked at Spark-UI.
I know setting up like this while creation of spark object, it's working well.

val spark = SparkSession.builder().appName(APP_NAME)
.config("spark.network.timeout", "1500s")
.config("spark.broadcast.compress", "true")
.config("spark.sql.broadcastTimeout", "36000")
.getOrCreate()

import spark.implicits._

Can someone please throw light?
Reply | Threaded
Open this post in threaded view
|

Re: Spark - configuration setting doesn't work

hemant singh
You should add the configurations while creating the session, I don’t think you can override it once the session is created. Few are though.

Thanks,
Hemant

On Sun, 27 Oct 2019 at 11:02 AM, Chetan Khatri <[hidden email]> wrote:
Could someone please help me.

On Thu, Oct 17, 2019 at 7:29 PM Chetan Khatri <[hidden email]> wrote:
Hi Users,

I am setting spark configuration in below way;

val spark = SparkSession.builder().appName(APP_NAME).getOrCreate()

spark.conf.set("spark.speculation", "false")
spark.conf.set("spark.broadcast.compress", "true")
spark.conf.set("spark.sql.broadcastTimeout", "36000")
spark.conf.set("spark.network.timeout", "2500s")
spark.conf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
spark.conf.set("spark.driver.memory", "10g")
spark.conf.set("spark.executor.memory", "10g")

import spark.implicits._

and submitting spark job with spark - submit. but none of the above configuration is
getting reflected to the job, I have checked at Spark-UI.
I know setting up like this while creation of spark object, it's working well.

val spark = SparkSession.builder().appName(APP_NAME)
.config("spark.network.timeout", "1500s")
.config("spark.broadcast.compress", "true")
.config("spark.sql.broadcastTimeout", "36000")
.getOrCreate()

import spark.implicits._

Can someone please throw light?
Reply | Threaded
Open this post in threaded view
|

Re: Spark - configuration setting doesn't work

Chetan Khatri
Ok, thanks. I wanted to confirm that.

On Sun, Oct 27, 2019 at 12:55 PM hemant singh <[hidden email]> wrote:
You should add the configurations while creating the session, I don’t think you can override it once the session is created. Few are though.

Thanks,
Hemant

On Sun, 27 Oct 2019 at 11:02 AM, Chetan Khatri <[hidden email]> wrote:
Could someone please help me.

On Thu, Oct 17, 2019 at 7:29 PM Chetan Khatri <[hidden email]> wrote:
Hi Users,

I am setting spark configuration in below way;

val spark = SparkSession.builder().appName(APP_NAME).getOrCreate()

spark.conf.set("spark.speculation", "false")
spark.conf.set("spark.broadcast.compress", "true")
spark.conf.set("spark.sql.broadcastTimeout", "36000")
spark.conf.set("spark.network.timeout", "2500s")
spark.conf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
spark.conf.set("spark.driver.memory", "10g")
spark.conf.set("spark.executor.memory", "10g")

import spark.implicits._

and submitting spark job with spark - submit. but none of the above configuration is
getting reflected to the job, I have checked at Spark-UI.
I know setting up like this while creation of spark object, it's working well.

val spark = SparkSession.builder().appName(APP_NAME)
.config("spark.network.timeout", "1500s")
.config("spark.broadcast.compress", "true")
.config("spark.sql.broadcastTimeout", "36000")
.getOrCreate()

import spark.implicits._

Can someone please throw light?