Loading a spark dataframe column into T-Digest using java

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Loading a spark dataframe column into T-Digest using java

Himasha de Silva
Hi, 

I want to load a spark dataframe column into T-Digest using java to calculate quantile values. I write this code to do this, but it's giving zero for size of tdigest. values are not added to tDigest. 

Reply | Threaded
Open this post in threaded view
|

Re: Loading a spark dataframe column into T-Digest using java

Marcelo Vanzin
The closure in your "foreach" loop runs in a remote executor, no the
local JVM, so it's updating its own copy of the t-digest instance. The
one on the driver side is never touched.

On Sun, Dec 10, 2017 at 10:27 PM, Himasha de Silva <[hidden email]> wrote:

> Hi,
>
> I want to load a spark dataframe column into T-Digest using java to
> calculate quantile values. I write this code to do this, but it's giving
> zero for size of tdigest. values are not added to tDigest.
>
> my code - https://gist.github.com/anonymous/1f2e382fdda002580154b5c43fbe9b3a
>
> Thank you.
>
> Himasha De Silva
>
>



--
Marcelo

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]