Spark Job Fails with Unknown Error writing to S3 from AWS EMR

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Spark Job Fails with Unknown Error writing to S3 from AWS EMR

koti reddy
Hi,

Can someone help to resolve this issue?
Thank you in advance.

Error logs :
java.io.EOFException: Unexpected EOF while trying to read response from server
	at org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:402)
	at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:213)
	at org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1073)
20/07/22 22:43:37 WARN DataStreamer: Error Recovery for BP-439833631-172.19.222.143-1595381416559:blk_1073742309_1498 in pipeline [DatanodeInfoWithStorage[172.19.222.182:50010,DS-7783002b-d57a-43a3-9d91-9934e2d063f8,DISK], DatanodeInfoWithStorage[172.19.223.27:50010,DS-1bc8cea7-9c28-4869-aada-55b7d0b0680c,DISK], DatanodeInfoWithStorage[172.19.223.199:50010,DS-880d5121-16a2-465e-ad20-ca99f4287770,DISK]]: datanode 0(DatanodeInfoWithStorage[172.19.222.182:50010,DS-7783002b-d57a-43a3-9d91-9934e2d063f8,DISK]) is bad.
20/07/22 22:44:58 WARN DataStreamer: Exception for BP-439833631-172.19.222.143-1595381416559:blk_1073742309_1499
java.io.EOFException: Unexpected EOF while trying to read response from server
	at org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:402)
	at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:213)
	at org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1073)
20/07/22 22:44:58 WARN DataStreamer: Error Recovery for BP-439833631-172.19.222.143-1595381416559:blk_1073742309_1499 in pipeline [DatanodeInfoWithStorage[172.19.223.27:50010,DS-1bc8cea7-9c28-4869-aada-55b7d0b0680c,DISK], DatanodeInfoWithStorage[172.19.223.199:50010,DS-880d5121-16a2-465e-ad20-ca99f4287770,DISK], DatanodeInfoWithStorage[172.19.222.180:50010,DS-40b1c81b-18d1-4d8d-ab49-11904f3dd23c,DISK]]: datanode 0(DatanodeInfoWithStorage[172.19.223.27:50010,DS-1bc8cea7-9c28-4869-aada-55b7d0b0680c,DISK]) is bad.
20/07/22 22:47:00 WARN DataStreamer: Exception for BP-439833631-172.19.222.143-1595381416559:blk_1073742309_1500
java.io.EOFException: Unexpected EOF while trying to read response from server
	at org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:402)
	at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:213)
	at org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1073)
20/07/22 22:47:00 WARN DataStreamer: Error Recovery for BP-439833631-172.19.222.143-1595381416559:blk_1073742309_1500 in pipeline [DatanodeInfoWithStorage[172.19.223.199:50010,DS-880d5121-16a2-465e-ad20-ca99f4287770,DISK], DatanodeInfoWithStorage[172.19.222.180:50010,DS-40b1c81b-18d1-4d8d-ab49-11904f3dd23c,DISK], DatanodeInfoWithStorage[172.19.223.55:50010,DS-3e5dc677-cd1d-49fc-b50a-4b058ae298aa,DISK]]: datanode 0(DatanodeInfoWithStorage[172.19.223.199:50010,DS-880d5121-16a2-465e-ad20-ca99f4287770,DISK]) is bad.
20/07/22 22:47:03 INFO MemoryStore: Block broadcast_4 stored as values in memory (estimated size 18.0 GB, free 24.5 GB)
--
Thanks,
Koti Reddy Nusum,
+1-(660) 541-5623.

Reply | Threaded
Open this post in threaded view
|

Re: Spark Job Fails with Unknown Error writing to S3 from AWS EMR

Shriraj Bhardwaj
We faced this similar situation with jre 8u262 try reverting back...

On Thu, Jul 23, 2020, 5:18 AM koti reddy <[hidden email]> wrote:
Hi,

Can someone help to resolve this issue?
Thank you in advance.

Error logs :
java.io.EOFException: Unexpected EOF while trying to read response from server
	at org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:402)
	at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:213)
	at org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1073)
20/07/22 22:43:37 WARN DataStreamer: Error Recovery for BP-439833631-172.19.222.143-1595381416559:blk_1073742309_1498 in pipeline [DatanodeInfoWithStorage[172.19.222.182:50010,DS-7783002b-d57a-43a3-9d91-9934e2d063f8,DISK], DatanodeInfoWithStorage[172.19.223.27:50010,DS-1bc8cea7-9c28-4869-aada-55b7d0b0680c,DISK], DatanodeInfoWithStorage[172.19.223.199:50010,DS-880d5121-16a2-465e-ad20-ca99f4287770,DISK]]: datanode 0(DatanodeInfoWithStorage[172.19.222.182:50010,DS-7783002b-d57a-43a3-9d91-9934e2d063f8,DISK]) is bad.
20/07/22 22:44:58 WARN DataStreamer: Exception for BP-439833631-172.19.222.143-1595381416559:blk_1073742309_1499
java.io.EOFException: Unexpected EOF while trying to read response from server
	at org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:402)
	at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:213)
	at org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1073)
20/07/22 22:44:58 WARN DataStreamer: Error Recovery for BP-439833631-172.19.222.143-1595381416559:blk_1073742309_1499 in pipeline [DatanodeInfoWithStorage[172.19.223.27:50010,DS-1bc8cea7-9c28-4869-aada-55b7d0b0680c,DISK], DatanodeInfoWithStorage[172.19.223.199:50010,DS-880d5121-16a2-465e-ad20-ca99f4287770,DISK], DatanodeInfoWithStorage[172.19.222.180:50010,DS-40b1c81b-18d1-4d8d-ab49-11904f3dd23c,DISK]]: datanode 0(DatanodeInfoWithStorage[172.19.223.27:50010,DS-1bc8cea7-9c28-4869-aada-55b7d0b0680c,DISK]) is bad.
20/07/22 22:47:00 WARN DataStreamer: Exception for BP-439833631-172.19.222.143-1595381416559:blk_1073742309_1500
java.io.EOFException: Unexpected EOF while trying to read response from server
	at org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:402)
	at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:213)
	at org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1073)
20/07/22 22:47:00 WARN DataStreamer: Error Recovery for BP-439833631-172.19.222.143-1595381416559:blk_1073742309_1500 in pipeline [DatanodeInfoWithStorage[172.19.223.199:50010,DS-880d5121-16a2-465e-ad20-ca99f4287770,DISK], DatanodeInfoWithStorage[172.19.222.180:50010,DS-40b1c81b-18d1-4d8d-ab49-11904f3dd23c,DISK], DatanodeInfoWithStorage[172.19.223.55:50010,DS-3e5dc677-cd1d-49fc-b50a-4b058ae298aa,DISK]]: datanode 0(DatanodeInfoWithStorage[172.19.223.199:50010,DS-880d5121-16a2-465e-ad20-ca99f4287770,DISK]) is bad.
20/07/22 22:47:03 INFO MemoryStore: Block broadcast_4 stored as values in memory (estimated size 18.0 GB, free 24.5 GB)
--
Thanks,
Koti Reddy Nusum,
+1-(660) 541-5623.