accumulator duration

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

accumulator duration


How long are accumulators held in memory?


For example, if I run Spark non stop, and my data has its own timestamp on which I aggregate AND some of the data comes delayed 24hrs for a previous aggregation done 24hrs ago, can I still use accumulators? Or will they be garbage collected due to space… or are they saved somewhere when they are GC-ed.