Given events with start and end times, how to count the number of simultaneous events using Spark?

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Given events with start and end times, how to count the number of simultaneous events using Spark?

newroyker
The problem statement and an approach to solve it using windows is described here:


Looking for more elegant/performant solutions, if they exist. TIA !
Reply | Threaded
Open this post in threaded view
|

Re: Given events with start and end times, how to count the number of simultaneous events using Spark?

kathleen li
You can use Spark sql window  function , something like
df.createOrReplaceTempView(“dfv”)
Select count(eventid) over ( partition by start_time, end_time orderly start_time) from  dfv

Sent from my iPhone

On Sep 26, 2018, at 11:32 AM, Debajyoti Roy <[hidden email]> wrote:

The problem statement and an approach to solve it using windows is described here:


Looking for more elegant/performant solutions, if they exist. TIA !