[pyspark 2.3+] broadcast timeout

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

[pyspark 2.3+] broadcast timeout

Hi All,

All of a sudden recently we discovered that all of our auto broadcasts have been timing out, this started happening in our static cloudera cluster as well as databricks. Data has not changed much. Has anyone seen anything like this before? Any suggestions other than increasing the timeout period or shutting off broadcast completely by setting the auto broadcast property to -1?


Rishi Shah