Sounds like Structured streaming with foreach, can only run on one executor

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Sounds like Structured streaming with foreach, can only run on one executor

Mich Talebzadeh
Hi,

When I use foreachBatch is Spark structured streaming, yarn mode works fine.

When one switches to foreach mode (row by row processing), this effectively runs in local mode on a single JVM. It seems to crash when running in a distributed mode. That is my experience.

Can someone else please verify this independently?

Cheers




LinkedIn  https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

 



Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.

 

Reply | Threaded
Open this post in threaded view
|

Re: Sounds like Structured streaming with foreach, can only run on one executor

srowen
That should not be the case. See https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#using-foreach-and-foreachbatch
Maybe you are calling .foreach on some Scala object inadvertently.

On Tue, Mar 9, 2021 at 4:41 PM Mich Talebzadeh <[hidden email]> wrote:
Hi,

When I use foreachBatch is Spark structured streaming, yarn mode works fine.

When one switches to foreach mode (row by row processing), this effectively runs in local mode on a single JVM. It seems to crash when running in a distributed mode. That is my experience.

Can someone else please verify this independently?

Cheers




LinkedIn  https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

 



Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.

 

Reply | Threaded
Open this post in threaded view
|

Re: Sounds like Structured streaming with foreach, can only run on one executor

Mich Talebzadeh
Thanks Sean,

I am using PySpark. There seems to be some reports on foreach usage with local mode back on the 3rd March. For example, see

"Spark structured streaming seems to work on local mode only"


I believe the thread owner was reporting on foreach case not foreachBatch.

cheers


Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.

 



On Tue, 9 Mar 2021 at 22:51, Sean Owen <[hidden email]> wrote:
That should not be the case. See https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#using-foreach-and-foreachbatch
Maybe you are calling .foreach on some Scala object inadvertently.

On Tue, Mar 9, 2021 at 4:41 PM Mich Talebzadeh <[hidden email]> wrote:
Hi,

When I use foreachBatch is Spark structured streaming, yarn mode works fine.

When one switches to foreach mode (row by row processing), this effectively runs in local mode on a single JVM. It seems to crash when running in a distributed mode. That is my experience.

Can someone else please verify this independently?

Cheers




LinkedIn  https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

 



Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.