She's now been in the queue for 13s (and the total users in the queue are now 43).
I need to keep track of two things:
(1) Since many of these are coming in from many users, I want to know, over some time period, how many of them are in the queue at any point,
(2) If Ms green.th was first seen at 13:04:04 and then at 13:04:05, she's been in the queue for 1 second (ignoring the ms).
How does one go about computing these sorts of more complex things in Spark Streaming? Would one have to keep track of her first-seen-time in a column and then do a diff the next time she's seen? With append / update mode, how does one begin doing this sort of thing?
Re: Keeping track of how long something has been in a queue
You may want to google around "session window" and "duration", and check whether the concept fits your requirements. Probably adding some custom logic on top of the session window would work for you, which requires you to implement a custom function for flatMapGroupsWithState.
Hope this helps.
Jungtaek Lim (HeartSaVioR)
On Fri, Sep 4, 2020 at 11:21 PM Hamish Whittal <[hidden email]> wrote:
Sorry, I moved a paragraph,
(2) If Ms green.th was first seen at 13:04:04, then at 13:04:05 and finally at 13:04:17, she's been in the queue for 13 seconds (ignoring the ms).