Spark Logging

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Spark Logging

srobertjames
How can I write to Spark's logs from my client code?
What are the options to view those logs?
Besides the Web console, is there a way to read and grep the file?
Reply | Threaded
Open this post in threaded view
|

Re: Spark Logging

Andrew Or-2
You can import org.apache.spark.Logging, and use logInfo, logWarning etc.
Besides viewing them from the Web console, the location of the logs can be found under $SPARK_HOME/logs, on both the driver and executor machines. (If you are on YARN, these logs are located elsewhere, however.)


2014-06-10 11:39 GMT-07:00 Robert James <[hidden email]>:
How can I write to Spark's logs from my client code?
What are the options to view those logs?
Besides the Web console, is there a way to read and grep the file?

Reply | Threaded
Open this post in threaded view
|

Re: Spark Logging

coderxiang
In reply to this post by srobertjames
By default, the logs are available at `/tmp/spark-events`. You can specify the log directory via spark.eventLog.dir, see this configuration page.
Reply | Threaded
Open this post in threaded view
|

Re: Spark Logging

Surendranauth Hiraman
Event logs are different from writing using a logger, like log4j. The event logs are the type of data showing up in the history server.

For my team, we use com.typesafe.scalalogging.slf4j.Logging. Our logs show up in /etc/spark/work/<app-id>/<executor-id>/stderr and stdout.

All of our logging seems to show up in stderr.

-Suren




On Tue, Jun 10, 2014 at 2:56 PM, coderxiang <[hidden email]> wrote:
By default, the logs are available at `/tmp/spark-events`. You can specify
the log directory via spark.eventLog.dir, see  this configuration page
<http://spark.apache.org/docs/latest/configuration.html>  .



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Logging-tp7340p7343.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.



--
                                                            
SUREN HIRAMAN, VP TECHNOLOGY
Velos
Accelerating Machine Learning

440 NINTH AVENUE, 11TH FLOOR
NEW YORK, NY 10001
O: (917) 525-2466 ext. 105
F: 646.349.4063
E: [hidden email]elos.io
W: www.velos.io