spark executor/driver log files management

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

spark executor/driver log files management

Sourav Chandra
Hi,

I have few questions regarding log file management in spark:

1. Currently I did not find any way to modify the lof file name for executor/drivers). Its hardcoded as stdout and stderr. Also there is no log rotation.

In case of streaming application this will grow forever and become unmanageable. Is there any way to overcome this?

Thanks,
--

Sourav Chandra

Senior Software Engineer

· · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · ·

[hidden email]

o: +91 80 4121 8723

m: +91 988 699 3746

skype: sourav.chandra

Livestream

"Ajmera Summit", First Floor, #3/D, 68 Ward, 3rd Cross, 7th C Main, 3rd Block, Koramangala Industrial Area,

Bangalore 560034

www.livestream.com

Reply | Threaded
Open this post in threaded view
|

Re: spark executor/driver log files management

Tathagata Das
You can use RollingFileAppenders in log4j.properties. 

You can have other scripts delete old logs. 

TD


On Mon, Mar 24, 2014 at 12:20 AM, Sourav Chandra <[hidden email]> wrote:
Hi,

I have few questions regarding log file management in spark:

1. Currently I did not find any way to modify the lof file name for executor/drivers). Its hardcoded as stdout and stderr. Also there is no log rotation.

In case of streaming application this will grow forever and become unmanageable. Is there any way to overcome this?

Thanks,
--

Sourav Chandra

Senior Software Engineer

· · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · ·

[hidden email]

o: <a href="tel:%2B91%2080%204121%208723" value="+918041218723" target="_blank">+91 80 4121 8723

m: <a href="tel:%2B91%20988%20699%203746" value="+919886993746" target="_blank">+91 988 699 3746

skype: sourav.chandra

Livestream

"Ajmera Summit", First Floor, #3/D, 68 Ward, 3rd Cross, 7th C Main, 3rd Block, Koramangala Industrial Area,

Bangalore 560034

www.livestream.com


Reply | Threaded
Open this post in threaded view
|

Re: spark executor/driver log files management

Sourav Chandra
Hi TD,

I thought about that but was not sure whether this will have any impact in spark UI/ Executor runner as it redirects stream to stderr/stdout. But ideally it should not as it will fetch the log record from stderr file (which is latest)..

Is my understanding correct?

Thanks,
Sourav


On Tue, Mar 25, 2014 at 3:26 AM, Tathagata Das <[hidden email]> wrote:
You can use RollingFileAppenders in log4j.properties. 

You can have other scripts delete old logs. 

TD


On Mon, Mar 24, 2014 at 12:20 AM, Sourav Chandra <[hidden email]> wrote:
Hi,

I have few questions regarding log file management in spark:

1. Currently I did not find any way to modify the lof file name for executor/drivers). Its hardcoded as stdout and stderr. Also there is no log rotation.

In case of streaming application this will grow forever and become unmanageable. Is there any way to overcome this?

Thanks,
--

Sourav Chandra

Senior Software Engineer

· · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · ·

[hidden email]

o: <a href="tel:%2B91%2080%204121%208723" value="+918041218723" target="_blank">+91 80 4121 8723

m: <a href="tel:%2B91%20988%20699%203746" value="+919886993746" target="_blank">+91 988 699 3746

skype: sourav.chandra

Livestream

"Ajmera Summit", First Floor, #3/D, 68 Ward, 3rd Cross, 7th C Main, 3rd Block, Koramangala Industrial Area,

Bangalore 560034

www.livestream.com





--

Sourav Chandra

Senior Software Engineer

· · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · ·

[hidden email]

o: +91 80 4121 8723

m: +91 988 699 3746

skype: sourav.chandra

Livestream

"Ajmera Summit", First Floor, #3/D, 68 Ward, 3rd Cross, 7th C Main, 3rd Block, Koramangala Industrial Area,

Bangalore 560034

www.livestream.com

Reply | Threaded
Open this post in threaded view
|

Re: spark executor/driver log files management

Tathagata Das
The logs from the executor are redirected to stdout only because there is a default log4j.properties that is configured to do so. If you put your log4j.properties with rolling file appender in the classpath (refer to Spark docs for that), all the logs will get redirected to a separate files that will get rolled over. So even though there will be a stdout, nothing much would get printed in that. Though the side effect of this modification is that you wont be able to see the logs from the Spark web UI as that only shows stdout.




On Mon, Mar 24, 2014 at 10:48 PM, Sourav Chandra <[hidden email]> wrote:
Hi TD,

I thought about that but was not sure whether this will have any impact in spark UI/ Executor runner as it redirects stream to stderr/stdout. But ideally it should not as it will fetch the log record from stderr file (which is latest)..

Is my understanding correct?

Thanks,
Sourav


On Tue, Mar 25, 2014 at 3:26 AM, Tathagata Das <[hidden email]> wrote:
You can use RollingFileAppenders in log4j.properties. 

You can have other scripts delete old logs. 

TD


On Mon, Mar 24, 2014 at 12:20 AM, Sourav Chandra <[hidden email]> wrote:
Hi,

I have few questions regarding log file management in spark:

1. Currently I did not find any way to modify the lof file name for executor/drivers). Its hardcoded as stdout and stderr. Also there is no log rotation.

In case of streaming application this will grow forever and become unmanageable. Is there any way to overcome this?

Thanks,
--

Sourav Chandra

Senior Software Engineer

· · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · ·

[hidden email]

o: <a href="tel:%2B91%2080%204121%208723" value="+918041218723" target="_blank">+91 80 4121 8723

m: <a href="tel:%2B91%20988%20699%203746" value="+919886993746" target="_blank">+91 988 699 3746

skype: sourav.chandra

Livestream

"Ajmera Summit", First Floor, #3/D, 68 Ward, 3rd Cross, 7th C Main, 3rd Block, Koramangala Industrial Area,

Bangalore 560034

www.livestream.com





--

Sourav Chandra

Senior Software Engineer

· · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · ·

[hidden email]

o: <a href="tel:%2B91%2080%204121%208723" value="+918041218723" target="_blank">+91 80 4121 8723

m: <a href="tel:%2B91%20988%20699%203746" value="+919886993746" target="_blank">+91 988 699 3746

skype: sourav.chandra

Livestream

"Ajmera Summit", First Floor, #3/D, 68 Ward, 3rd Cross, 7th C Main, 3rd Block, Koramangala Industrial Area,

Bangalore 560034

www.livestream.com