[K8S] Driver and Executor Logging

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

[K8S] Driver and Executor Logging

Rohit Menon
Hello All,

We are trying to use a custom appender for Spark driver and executor pods. However, the changes to log4j.properties file in the spark container image are not taking effect. We even tried simpler changes like changing the logging level to DEBUG. 

Has anyone run into similar issues? or successfully changed logging properties for spark on k8s?

Thanks,
Rohit
Reply | Threaded
Open this post in threaded view
|

RE: [K8S] Driver and Executor Logging

Garlapati, Suryanarayana (Nokia - IN/Bangalore)

Hi,

Provide the following options in spark-defaults.conf and make sure the log4j.properties file is available in driver and executor:

 

spark.driver.extraJavaOptions -Dlog4j.configuration=file:<path>/log4j.properties

spark.executor.extraJavaOptions -Dlog4j.configuration=file: <path>/log4j.properties

 

 

Regards

Surya

 

From: Rohit Menon <[hidden email]>
Sent: Friday, September 7, 2018 11:02 PM
To: [hidden email]
Subject: [K8S] Driver and Executor Logging

 

Hello All,

 

We are trying to use a custom appender for Spark driver and executor pods. However, the changes to log4j.properties file in the spark container image are not taking effect. We even tried simpler changes like changing the logging level to DEBUG. 

 

Has anyone run into similar issues? or successfully changed logging properties for spark on k8s?

 

Thanks,

Rohit