groupBy question

classic Classic list List threaded Threaded
3 messages Options
SK
Reply | Threaded
Open this post in threaded view
|

groupBy question

SK
After doing a groupBy operation, I have the following result:

 val res =
("ID1",ArrayBuffer((1458046000001,"ID1","japan")))
("ID3",ArrayBuffer((1458650800000,"ID3","canada"), (1458996400000,"ID3","china")))
("ID2",ArrayBuffer((1457527600000,"ID2","usa"), (1459342000000,"ID2","usa")))

Now I need to output for each group, the size of each group and the max of the first field, which is a timestamp.
So, I tried the following:

1) res.map(group => (group._2.size, group._2._1.max))
But I got an error : value _1 is not a member of Iterable[(Long, String, String)]

2) I also tried: res.map(group => (group._2.size, group._2[1].max)), but got an error for that as well.

What is the right way to get the max of the timestamp field (the first field in the ArrayBuffer) for each group?


thanks.
Reply | Threaded
Open this post in threaded view
|

Re: groupBy question

coderxiang
res.map(group => (group._2.size, group._2.map(_._1).max))


On Tue, Jun 10, 2014 at 6:10 PM, SK <[hidden email]> wrote:
After doing a groupBy operation, I have the following result:

 val res =
("ID1",ArrayBuffer((1458046000001,"ID1","japan")))
("ID3",ArrayBuffer((1458650800000,"ID3","canada"),
(1458996400000,"ID3","china")))
("ID2",ArrayBuffer((1457527600000,"ID2","usa"),
(1459342000000,"ID2","usa")))

Now I need to output for each group, the size of each group and the max of
the first field, which is a timestamp.
So, I tried the following:

1) res.map(group => (group._2.size, group._2._1.max))
But I got an error : value _1 is not a member of Iterable[(Long, String,
String)]

2) I also tried: res.map(group => (group._2.size, group._2[1].max)), but got
an error for that as well.

What is the right way to get the max of the timestamp field (the first field
in the ArrayBuffer) for each group?


thanks.



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/groupBy-question-tp7357.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

SK
Reply | Threaded
Open this post in threaded view
|

Re: groupBy question

SK
Great, thanks!