Spark Event Log Forwarding and Offset Tracking

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Spark Event Log Forwarding and Offset Tracking

raymond.tan
Hello here, I am new to spark and am trying to add some monitoring for spark applications specifically to handle the below situations - 1 - Forwarding Spark Event Logs to identify critical events like job start, executor failures, job failures etc to ElasticSearch via log4j. However I could not find any way to foward event log via log4j configurations. Is there any other recommended approach to track these application events? 2 - For Spark streaming jobs, is there any way to identify that data from Kafka is not consumed for whatever reason, or the offsets are not progressing as expected and also forward that to ElasticSearch via log4j for monitoring Thanks, Raymond

Sent from the Apache Spark User List mailing list archive at Nabble.com.
Reply | Threaded
Open this post in threaded view
|

Re: Spark Event Log Forwarding and Offset Tracking

Jacek Laskowski
Hi,

> Forwarding Spark Event Logs to identify critical events like job start, executor failures, job failures etc to ElasticSearch via log4j. However I could not find any way to foward event log via log4j configurations. Is there any other recommended approach to track these application events?


> 2 - For Spark streaming jobs, is there any way to identify that data from Kafka is not consumed for whatever reason, or the offsets are not progressing as expected and also forward that to ElasticSearch via log4j for monitoring

Think SparkListener API would help here too.

On Wed, Jan 13, 2021 at 5:15 PM raymond.tan <[hidden email]> wrote:
Hello here, I am new to spark and am trying to add some monitoring for spark applications specifically to handle the below situations - 1 - Forwarding Spark Event Logs to identify critical events like job start, executor failures, job failures etc to ElasticSearch via log4j. However I could not find any way to foward event log via log4j configurations. Is there any other recommended approach to track these application events? 2 - For Spark streaming jobs, is there any way to identify that data from Kafka is not consumed for whatever reason, or the offsets are not progressing as expected and also forward that to ElasticSearch via log4j for monitoring Thanks, Raymond

Sent from the Apache Spark User List mailing list archive at Nabble.com.