Enhanced Logging on Pyspark

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Enhanced Logging on Pyspark

aazout
This post has NOT been accepted by the mailing list yet.
I get the "org.apache.spark.SparkException: Python worker exited unexpectedly (crashed)" message on my Spark cluster installed on EC2. I imagine this is because I have not installed all required python dependencies on the worker node.

It would be good to get more detailed python logging to be able to debug pyspark problems.

Thanks!
CEO / Velos (velos.io)