EOF Exception Spark Structured Streams - Kubernetes

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

EOF Exception Spark Structured Streams - Kubernetes

Sachit Murarka
Hi Users,

I am running Spark application on Kubernetes and getting the following exception in the driver pod. Though it is not affecting the output. 

This exception is coming every 5 minutes and this is a structured streaming job.

Could anyone please advise ?

21/01/29 06:33:15 WARN WatchConnectionManager: Exec Failurejava.io.EOFException at okio.RealBufferedSource.require(RealBufferedSource.java:61) at okio.RealBufferedSource.readByte(RealBufferedSource.java:74) at okhttp3.internal.ws.WebSocketReader.readHeader(WebSocketReader.java:117) at okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:101) at okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.java:274) at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:214) at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203) at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834)21/01/29 06:38:16 WARN WatchConnectionManager: Exec Failure


Kind Regards,
Sachit Murarka
Reply | Threaded
Open this post in threaded view
|

Re: EOF Exception Spark Structured Streams - Kubernetes

Sachit Murarka
Following is the related JIRA , Can someone pls check


I am using 3.0.1 , It says fixed in 3.0.0 and 3.1.0 . Could you please suggest what can be done to avoid this?

Kind Regards,
Sachit Murarka


On Sun, Jan 31, 2021 at 6:38 PM Sachit Murarka <[hidden email]> wrote:
Hi Users,

I am running Spark application on Kubernetes and getting the following exception in the driver pod. Though it is not affecting the output. 

This exception is coming every 5 minutes and this is a structured streaming job.

Could anyone please advise ?

21/01/29 06:33:15 WARN WatchConnectionManager: Exec Failurejava.io.EOFException at okio.RealBufferedSource.require(RealBufferedSource.java:61) at okio.RealBufferedSource.readByte(RealBufferedSource.java:74) at okhttp3.internal.ws.WebSocketReader.readHeader(WebSocketReader.java:117) at okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:101) at okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.java:274) at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:214) at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203) at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834)21/01/29 06:38:16 WARN WatchConnectionManager: Exec Failure


Kind Regards,
Sachit Murarka
Reply | Threaded
Open this post in threaded view
|

Re: EOF Exception Spark Structured Streams - Kubernetes

Prashant Sharma
Hi Sachit,

The fix verison on that JIRA says 3.0.2, so this fix is not yet released. Soon, there will be a 3.1.1 release, in the meantime you can try out the 3.1.1-rc which also has the fix and let us know your findings.

Thanks,


On Mon, Feb 1, 2021 at 10:24 AM Sachit Murarka <[hidden email]> wrote:
Following is the related JIRA , Can someone pls check


I am using 3.0.1 , It says fixed in 3.0.0 and 3.1.0 . Could you please suggest what can be done to avoid this?

Kind Regards,
Sachit Murarka


On Sun, Jan 31, 2021 at 6:38 PM Sachit Murarka <[hidden email]> wrote:
Hi Users,

I am running Spark application on Kubernetes and getting the following exception in the driver pod. Though it is not affecting the output. 

This exception is coming every 5 minutes and this is a structured streaming job.

Could anyone please advise ?

21/01/29 06:33:15 WARN WatchConnectionManager: Exec Failurejava.io.EOFException at okio.RealBufferedSource.require(RealBufferedSource.java:61) at okio.RealBufferedSource.readByte(RealBufferedSource.java:74) at okhttp3.internal.ws.WebSocketReader.readHeader(WebSocketReader.java:117) at okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:101) at okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.java:274) at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:214) at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203) at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834)21/01/29 06:38:16 WARN WatchConnectionManager: Exec Failure


Kind Regards,
Sachit Murarka
Reply | Threaded
Open this post in threaded view
|

Re: EOF Exception Spark Structured Streams - Kubernetes

Sachit Murarka
Hi Prashant, 

Yes I understand that will be available from Spark 3.0.2 . But I am using Spark3.0.1

Could you please suggest any way to mitigate this exception?

Will it cause any issues from this exception in the spark application?

Thanks
Sachit

On Mon, 1 Feb 2021, 13:57 Prashant Sharma, <[hidden email]> wrote:
Hi Sachit,

The fix verison on that JIRA says 3.0.2, so this fix is not yet released. Soon, there will be a 3.1.1 release, in the meantime you can try out the 3.1.1-rc which also has the fix and let us know your findings.

Thanks,


On Mon, Feb 1, 2021 at 10:24 AM Sachit Murarka <[hidden email]> wrote:
Following is the related JIRA , Can someone pls check


I am using 3.0.1 , It says fixed in 3.0.0 and 3.1.0 . Could you please suggest what can be done to avoid this?

Kind Regards,
Sachit Murarka


On Sun, Jan 31, 2021 at 6:38 PM Sachit Murarka <[hidden email]> wrote:
Hi Users,

I am running Spark application on Kubernetes and getting the following exception in the driver pod. Though it is not affecting the output. 

This exception is coming every 5 minutes and this is a structured streaming job.

Could anyone please advise ?

21/01/29 06:33:15 WARN WatchConnectionManager: Exec Failurejava.io.EOFException at okio.RealBufferedSource.require(RealBufferedSource.java:61) at okio.RealBufferedSource.readByte(RealBufferedSource.java:74) at okhttp3.internal.ws.WebSocketReader.readHeader(WebSocketReader.java:117) at okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:101) at okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.java:274) at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:214) at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203) at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834)21/01/29 06:38:16 WARN WatchConnectionManager: Exec Failure


Kind Regards,
Sachit Murarka