Problem understanding log message in SparkStreaming

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Problem understanding log message in SparkStreaming

nilmish
I wanted to know the meaning of the following log message when running a spark streaming job :

[spark-akka.actor.default-dispatcher-18] INFO  org.apache.spark.streaming.scheduler.JobScheduler - Total delay: 5.432 s for time 1401870454500 ms (execution: 0.593 s)

According to my understanding, total delay here means total end-to-end delay which is here 5.432 sec.

What is the meaning of execution : 0.593 ??

Is it the time taken for executing this particular query ?

PS : I am running a streaming job over a window of 5 mins and quering every 1.5 sec.