SparkR issue

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

SparkR issue

ayan guha
Hi

We are seeing some weird behaviour in Spark R. 

We created a R Dataframe with 600K records and 29 columns. Then we tried to convert R DF to SparkDF using 

df <- SparkR::createDataFrame(rdf)

from RStudio. It hanged, we had to kill the process after 1-2 hours. 

We also tried following:
df <- SparkR::createDataFrame(rdf, numPartition=4000)
df <- SparkR::createDataFrame(rdf, numPartition=300) 
df <- SparkR::createDataFrame(rdf, numPartition=10) 

Same result. Both scenarios seems RStudio is working and no trace of jobs in Spark Application Master view. 

Finally, we used this:

df <- SparkR::createDataFrame(rdf, schema=schema) , schema is a StructType.

This tool 25 mins to create the spark DF. However job did show up in Application Master view and it shows only 20-30 secs. Then where did rest of the time go?

Question:
1. Is this expected behavior? (I hope not). How should we speed up this bit?
2. We understand better options would be to read data from external sources, but we need this data to be generated for some simulation purpose. Whats possibly going wrong?


Best
Ayan
  


--
Best Regards,
Ayan Guha
Reply | Threaded
Open this post in threaded view
|

Re: SparkR issue

Felix Cheung
1 seems like its spending a lot of time in R (slicing the data I guess?) and not with Spark
2 could you write it into a csv file locally and then read it from Spark?

 

From: ayan guha <[hidden email]>
Sent: Monday, October 8, 2018 11:21 PM
To: user
Subject: SparkR issue
 
Hi

We are seeing some weird behaviour in Spark R. 

We created a R Dataframe with 600K records and 29 columns. Then we tried to convert R DF to SparkDF using 

df <- SparkR::createDataFrame(rdf)

from RStudio. It hanged, we had to kill the process after 1-2 hours. 

We also tried following:
df <- SparkR::createDataFrame(rdf, numPartition=4000)
df <- SparkR::createDataFrame(rdf, numPartition=300) 
df <- SparkR::createDataFrame(rdf, numPartition=10) 

Same result. Both scenarios seems RStudio is working and no trace of jobs in Spark Application Master view. 

Finally, we used this:

df <- SparkR::createDataFrame(rdf, schema=schema) , schema is a StructType.

This tool 25 mins to create the spark DF. However job did show up in Application Master view and it shows only 20-30 secs. Then where did rest of the time go?

Question:
1. Is this expected behavior? (I hope not). How should we speed up this bit?
2. We understand better options would be to read data from external sources, but we need this data to be generated for some simulation purpose. Whats possibly going wrong?


Best
Ayan
  


--
Best Regards,
Ayan Guha