possible cause: same TeraGen job sometimes slow and sometimes fast

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

possible cause: same TeraGen job sometimes slow and sometimes fast

Gil Vernik
I performed a series of TeraGen jobs via spark-submit ( each job generated equal size dataset into different S3 buckets )
I noticed that some jobs were fast and some were slow.

Slow jobs always had many log prints like
DEBUG TaskSchedulerImpl: parentName: , name: TaskSet_1.0, runningTasks: 1 ( or 2, etc.. )

Fast jobs always have few prints of those lines.

Can someone explain me, why the number of those debug prints are vary for different executions of the same job? The more i see those prints - so the job is slower.
Does someone experienced the same behavior?

Thanks
Gil.