Refresh broadcast variable when it isn't the value.

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Refresh broadcast variable when it isn't the value.

Guillermo Ortiz Fernández
Hello,

I want to set data in a broadcast (Map<key,value>) variable in Spark. Sometimes there are new data so I have to update/refresh the values but I'm not sure how I could do this.

My idea is to use accumulators like a flag when a cache error occurs, in this point I could read the data and reload the broadcast variable for the next microbach and finally reset the accumulator to 0.
I don't know if there are a better solution or others ideas to do this.

Has anyone faced this problem?