Is dataframe write blocking? what can be done for fair scheduler?

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Is dataframe write blocking? what can be done for fair scheduler?

ramannanda9@gmail.com
Hi,

I have noticed that in fair scheduler setting, if i block on dataframe write
to complete, using AwaitResult, the API call ends up returning, whereas that
is not what I intend to do as it can cause inconsistencies later in the
pipeline. Is there a way to make the dataframe write call blocking?


Regards,
Ramandeep Singh



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]