What do you pay attention to when validating Spark jobs?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

What do you pay attention to when validating Spark jobs?

Holden Karau
Hi Folks,

I'm working on updating a talk and I was wondering if any folks in the community wanted to share their best practices for validating your Spark jobs? Are there any counters folks have found useful for monitoring/validating your Spark jobs?

Cheers,

Holden :)

--
Reply | Threaded
Open this post in threaded view
|

Re: What do you pay attention to when validating Spark jobs?

lucas.gary@gmail.com
I don't think these will blow anyones minds but:

1) Row counts.  Most of our jobs 'recompute the world' nightly so we can expect to see fairly predictable row variances.
2) Rolling snapshots.  We can also expect that for some critical datasets we can compute a rolling average for important metrics (revenue, user count, etc).  We're just starting to investigate this.
3) Job timing:  Jobs should normally take about the same amount of time to execute (usually).  So we want to alert on things that finish too quickly (no data in the pipe) or things that take too long.

I'd like to get further into anomaly detection but haven't gotten there yet.

On 21 November 2017 at 15:34, Holden Karau <[hidden email]> wrote:
Hi Folks,

I'm working on updating a talk and I was wondering if any folks in the community wanted to share their best practices for validating your Spark jobs? Are there any counters folks have found useful for monitoring/validating your Spark jobs?

Cheers,

Holden :)

--