Async action in Dataframe

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Async action in Dataframe

JiaTao Tao
Hi all
As we all know, RDD has the operation: "collectAsync()", also submitJob can return a Future, but I cannot find the same thing in Dataset, anyone knows how I can archive this when using "Dataframe"?

Thanks a lot.
--


Regards!

Aron Tao

Reply | Threaded
Open this post in threaded view
|

Re: Async action in Dataframe

Jiaan Geng
RDD have not the method `collectAsync`.There exists a implicit conversion
from RDD to AsyncRDDActions in object RDD. The implicit conversion is :
  implicit def rddToAsyncRDDActions[T: ClassTag](rdd: RDD[T]):
AsyncRDDActions[T] = {
    new AsyncRDDActions(rdd)
  }
The method collect of RDD use the SparkContext.runJob,But the method
collectAsync of AsyncRDDActions use SparkContext.submitJob.
You can refer this difference to achieve this function.




--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re:Re: Async action in Dataframe

Jiaan Geng
I check the `collect` of `DataSet`, this method call the `collect` of `RDD` and apply `decodeUnsafeRows`.
So I think the function of the two `collect` is differenct.
The `collect` of `DataSet` is used for spark sql.
If you really want use `collectAsync`,please code following:
`df.rdd.collectAsync`






At 2018-12-24 11:36:14, "Jiaan Geng" <[hidden email]> wrote: >RDD have not the method `collectAsync`.There exists a implicit conversion >from RDD to AsyncRDDActions in object RDD. The implicit conversion is : > implicit def rddToAsyncRDDActions[T: ClassTag](rdd: RDD[T]): >AsyncRDDActions[T] = { > new AsyncRDDActions(rdd) > } >The method collect of RDD use the SparkContext.runJob,But the method >collectAsync of AsyncRDDActions use SparkContext.submitJob. >You can refer this difference to achieve this function. > > > > >-- >Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ > >--------------------------------------------------------------------- >To unsubscribe e-mail: [hidden email]