Quantcast

[RDDs and Dataframes] Equivalent expressions for RDD API

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[RDDs and Dataframes] Equivalent expressions for RDD API

Old-School
This post has NOT been accepted by the mailing list yet.
Hi,

I want to perform some simple transformations and check the execution time, under various configurations (e.g. number of cores being used, number of partitions etc). Since it is not possible to set the partitions of a dataframe , I guess that I should probably use RDDs.

I've got a dataset with 3 columns as shown below:

val data = file.map(line => line.split(" "))
              .filter(lines => lines.length == 3) // ignore first line
              .map(row => (row(0), row(1), row(2)))
              .toDF("ID", "word-ID", "count")
results in:

+------+------------+---------+
| ID     |  word-ID   |  count   |
+------+------------+---------+
|  15   |    87          |   151    |
|  20   |    19          |   398    |
|  15   |    19          |   21      |
|  180 |    90          |   190    |
+-------------------+---------+
So how can I turn the above into an RDD in order to use e.g. sc.parallelize(data, 10) and set the number of partitions to say 10?

Furthermore, I would also like to ask about the equivalent expression (using RDD API) for the following simple transformation:

data.select("word-ID", "count").groupBy("word-ID").agg(sum($"count").as("count")).show()



Thanks in advance
Loading...