No space left on device

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

No space left on device

Mina Aslani
Hi,

I get "No space left on device" error in my spark worker:

Error writing stream to file /usr/spark-2.2.0/work/app-......./0/stderr
java.io.IOException: No space left on device

In my spark cluster, I have one worker and one master. 
My program consumes stream of data from kafka and publishes the result into kafka. I set my RDD = null after I finish working, so that intermediate shuffle files are removed quickly.

How can I avoid "No space left on device"?

Best regards,
Mina
Reply | Threaded
Open this post in threaded view
|

Re: No space left on device

Chetan Khatri

Your hard drive don't have much space

On 18-Oct-2017 10:35 AM, "Mina Aslani" <[hidden email]> wrote:
Hi,

I get "No space left on device" error in my spark worker:

Error writing stream to file /usr/spark-2.2.0/work/app-......./0/stderr
java.io.IOException: No space left on device

In my spark cluster, I have one worker and one master. 
My program consumes stream of data from kafka and publishes the result into kafka. I set my RDD = null after I finish working, so that intermediate shuffle files are removed quickly.

How can I avoid "No space left on device"?

Best regards,
Mina
Reply | Threaded
Open this post in threaded view
|

Re: No space left on device

Chetan Khatri

Process data in micro batch

On 18-Oct-2017 10:36 AM, "Chetan Khatri" <[hidden email]> wrote:

Your hard drive don't have much space

On 18-Oct-2017 10:35 AM, "Mina Aslani" <[hidden email]> wrote:
Hi,

I get "No space left on device" error in my spark worker:

Error writing stream to file /usr/spark-2.2.0/work/app-......./0/stderr
java.io.IOException: No space left on device

In my spark cluster, I have one worker and one master. 
My program consumes stream of data from kafka and publishes the result into kafka. I set my RDD = null after I finish working, so that intermediate shuffle files are removed quickly.

How can I avoid "No space left on device"?

Best regards,
Mina
Reply | Threaded
Open this post in threaded view
|

Re: No space left on device

Imran Rajjad
In reply to this post by Mina Aslani
did you try calling rdd.unpersist()

On Wed, Oct 18, 2017 at 10:04 AM, Mina Aslani <[hidden email]> wrote:
Hi,

I get "No space left on device" error in my spark worker:

Error writing stream to file /usr/spark-2.2.0/work/app-......./0/stderr
java.io.IOException: No space left on device

In my spark cluster, I have one worker and one master. 
My program consumes stream of data from kafka and publishes the result into kafka. I set my RDD = null after I finish working, so that intermediate shuffle files are removed quickly.

How can I avoid "No space left on device"?

Best regards,
Mina



--
I.R
Reply | Threaded
Open this post in threaded view
|

Re: No space left on device

Mina Aslani
I have not tried rdd.unpersist(), I thought using rdd = null is the same, is it not?

On Wed, Oct 18, 2017 at 1:07 AM, Imran Rajjad <[hidden email]> wrote:
did you try calling rdd.unpersist()

On Wed, Oct 18, 2017 at 10:04 AM, Mina Aslani <[hidden email]> wrote:
Hi,

I get "No space left on device" error in my spark worker:

Error writing stream to file /usr/spark-2.2.0/work/app-......./0/stderr
java.io.IOException: No space left on device

In my spark cluster, I have one worker and one master. 
My program consumes stream of data from kafka and publishes the result into kafka. I set my RDD = null after I finish working, so that intermediate shuffle files are removed quickly.

How can I avoid "No space left on device"?

Best regards,
Mina



--
I.R

Reply | Threaded
Open this post in threaded view
|

Re: No space left on device

Imran Rajjad
don't think so. check out the documentation for this method

On Wed, Oct 18, 2017 at 10:11 AM, Mina Aslani <[hidden email]> wrote:
I have not tried rdd.unpersist(), I thought using rdd = null is the same, is it not?

On Wed, Oct 18, 2017 at 1:07 AM, Imran Rajjad <[hidden email]> wrote:
did you try calling rdd.unpersist()

On Wed, Oct 18, 2017 at 10:04 AM, Mina Aslani <[hidden email]> wrote:
Hi,

I get "No space left on device" error in my spark worker:

Error writing stream to file /usr/spark-2.2.0/work/app-......./0/stderr
java.io.IOException: No space left on device

In my spark cluster, I have one worker and one master. 
My program consumes stream of data from kafka and publishes the result into kafka. I set my RDD = null after I finish working, so that intermediate shuffle files are removed quickly.

How can I avoid "No space left on device"?

Best regards,
Mina



--
I.R




--
I.R