Is Dataframe write blocking?

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

Is Dataframe write blocking?

I have some futures setup to operate in stages, where I expect one stage to complete before another begins. I was hoping that dataframe write call is blocking, whereas the behavior i see is that the call returns before data is persisted. This can cause unintended consequences. I am also using fair scheduler so that independent jobs can run in parallel. 

Spark 2.3.1.

Ramandeep Singh