Stage or Tasks level logs missing

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

Stage or Tasks level logs missing

Nirav Patel
Currently there seems to be 3 places to check task level logs:
1) Using spark UI 
2) `yarn application log` 
3) log aggregation  on hdfs (if enabled)

All above only give you log at executor(container) level. However one executor can have multiple threads and each might be running part of different stages(stg1, stg2)) and within that different tasks(tid1, tid2...) . It's hard to track particular task activities  in Executor logs.
It'd be nice if:
1) mark all log entries with stageId followed by taskId
2) have a separate log file for each task (taskId)
3) have a separated log file for of stage level logs

If I missed something let me know