Dynamically changing maxOffsetsPerTrigger

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

Dynamically changing maxOffsetsPerTrigger

Something Something
Is there a way to dynamically modify value of 'maxOffsetsPerTrigger' while a Stateful Structured Streaming job is running?

We are thinking of auto-scaling our Spark cluster but if we don't modify the value of 'maxOffsetsPerTrigger' dynamically would adding more VMs to the cluster help? I don't think it would, would it?

In other words, if I add 2 new VMs to the cluster but value of 'maxOffsetsPerTrigger' is still the same would performance improve? I would think not. We would have to explicitly stop the job, add VMs & then restart the job after changing the value of 'maxOffsetsPerTrigger' - which defeats the purpose of Auto-scaling.

Please tell me if my understanding is not correct. Thanks.