[0.9.0] MEMORY_AND_DISK_SER not falling back to disk

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

[0.9.0] MEMORY_AND_DISK_SER not falling back to disk

Andrew Ash
// version 0.9.0

Hi Spark users,

My understanding of the MEMORY_AND_DISK_SER persistence level was that if an RDD could fit into memory then it would be left there (same as MEMORY_ONLY), and only if it was too big for memory would it spill to disk.  Here's how the docs describe it:

MEMORY_AND_DISK_SER Similar to MEMORY_ONLY_SER, but spill partitions that don't fit in memory to disk instead of recomputing them on the fly each time they're needed.



What I'm observing though is that really large RDDs are actually causing OOMs.  I'm not sure if this is a regression in 0.9.0 or if it has been this way for some time.

While I look through the source code, has anyone actually observed the correct spill to disk behavior rather than an OOM?

Thanks!
Andrew
Reply | Threaded
Open this post in threaded view
|

Re: [0.9.0] MEMORY_AND_DISK_SER not falling back to disk

Matei Zaharia
Administrator
This probably means that there’s not enough free memory for the “scratch” space used for computations, so we OOM before the Spark cache decides that it’s full and starts to spill stuff. Try reducing spark.storage.memoryFraction (default is 0.66, try 0.5).

Matei

On Feb 5, 2014, at 10:29 PM, Andrew Ash <[hidden email]> wrote:

// version 0.9.0

Hi Spark users,

My understanding of the MEMORY_AND_DISK_SER persistence level was that if an RDD could fit into memory then it would be left there (same as MEMORY_ONLY), and only if it was too big for memory would it spill to disk.  Here's how the docs describe it:

MEMORY_AND_DISK_SER Similar to MEMORY_ONLY_SER, but spill partitions that don't fit in memory to disk instead of recomputing them on the fly each time they're needed.



What I'm observing though is that really large RDDs are actually causing OOMs.  I'm not sure if this is a regression in 0.9.0 or if it has been this way for some time.

While I look through the source code, has anyone actually observed the correct spill to disk behavior rather than an OOM?

Thanks!
Andrew

Reply | Threaded
Open this post in threaded view
|

Re: [0.9.0] MEMORY_AND_DISK_SER not falling back to disk

Andrew Ash
I dropped down to 0.5 but still OOM'd, so sent it all the way to 0.1 and didn't get an OOM.  I could tune this some more to find where the cliff is, but this is a one-off job so now that it's completed I don't want to spend any more time tuning it.

Is there a reason that this value couldn't be dynamically adjusted in response to actual heap usage?

I can imagine a scenario where spending too much time in GC (descending into GC hell) drops the value a little to keep from OOM, or directly measuring how much of the heap is spent on this scratch space and adjusting appropriately.


On Sat, Feb 8, 2014 at 3:40 PM, Matei Zaharia <[hidden email]> wrote:
This probably means that there’s not enough free memory for the “scratch” space used for computations, so we OOM before the Spark cache decides that it’s full and starts to spill stuff. Try reducing spark.storage.memoryFraction (default is 0.66, try 0.5).

Matei

On Feb 5, 2014, at 10:29 PM, Andrew Ash <[hidden email]> wrote:

// version 0.9.0

Hi Spark users,

My understanding of the MEMORY_AND_DISK_SER persistence level was that if an RDD could fit into memory then it would be left there (same as MEMORY_ONLY), and only if it was too big for memory would it spill to disk.  Here's how the docs describe it:

MEMORY_AND_DISK_SER Similar to MEMORY_ONLY_SER, but spill partitions that don't fit in memory to disk instead of recomputing them on the fly each time they're needed.



What I'm observing though is that really large RDDs are actually causing OOMs.  I'm not sure if this is a regression in 0.9.0 or if it has been this way for some time.

While I look through the source code, has anyone actually observed the correct spill to disk behavior rather than an OOM?

Thanks!
Andrew


Reply | Threaded
Open this post in threaded view
|

Re: [0.9.0] MEMORY_AND_DISK_SER not falling back to disk

Rafal Kwasny
Hi Everyone,
Maybe it's a good time to reevaluate off-heap storage for RDD's with custom allocator?

On a few occasions recently I had to lower both spark.storage.memoryFraction and spark.shuffle.memoryFraction
spark.shuffle.spill helps a bit with large scale reduces

Also it could be you're hitting: https://github.com/apache/incubator-spark/pull/180

/Rafal


Andrew Ash wrote:
I dropped down to 0.5 but still OOM'd, so sent it all the way to 0.1 and didn't get an OOM.  I could tune this some more to find where the cliff is, but this is a one-off job so now that it's completed I don't want to spend any more time tuning it.

Is there a reason that this value couldn't be dynamically adjusted in response to actual heap usage?

I can imagine a scenario where spending too much time in GC (descending into GC hell) drops the value a little to keep from OOM, or directly measuring how much of the heap is spent on this scratch space and adjusting appropriately.


On Sat, Feb 8, 2014 at 3:40 PM, Matei Zaharia <[hidden email]> wrote:
This probably means that there’s not enough free memory for the “scratch” space used for computations, so we OOM before the Spark cache decides that it’s full and starts to spill stuff. Try reducing spark.storage.memoryFraction (default is 0.66, try 0.5).

Matei

On Feb 5, 2014, at 10:29 PM, Andrew Ash <[hidden email]> wrote:

// version 0.9.0

Hi Spark users,

My understanding of the MEMORY_AND_DISK_SER persistence level was that if an RDD could fit into memory then it would be left there (same as MEMORY_ONLY), and only if it was too big for memory would it spill to disk.  Here's how the docs describe it:

MEMORY_AND_DISK_SER Similar to MEMORY_ONLY_SER, but spill partitions that don't fit in memory to disk instead of recomputing them on the fly each time they're needed.



What I'm observing though is that really large RDDs are actually causing OOMs.  I'm not sure if this is a regression in 0.9.0 or if it has been this way for some time.

While I look through the source code, has anyone actually observed the correct spill to disk behavior rather than an OOM?

Thanks!
Andrew



Reply | Threaded
Open this post in threaded view
|

Re: [0.9.0] MEMORY_AND_DISK_SER not falling back to disk

Andrew Ash
My understanding of off-heap storage was that you'd still need to get those JVM objects on-heap in order to actually use them with map, filter, etc.  Would we be trading CPU time to get memory efficiency if we went down the off-heap storage route?  I'm not sure what discussions have already happened here and what kind of implementation we're talking about.




On Mon, Feb 10, 2014 at 2:28 AM, Rafal Kwasny <[hidden email]> wrote:
Hi Everyone,
Maybe it's a good time to reevaluate off-heap storage for RDD's with custom allocator?

On a few occasions recently I had to lower both spark.storage.memoryFraction and spark.shuffle.memoryFraction
spark.shuffle.spill helps a bit with large scale reduces

Also it could be you're hitting: https://github.com/apache/incubator-spark/pull/180

/Rafal



Andrew Ash wrote:
I dropped down to 0.5 but still OOM'd, so sent it all the way to 0.1 and didn't get an OOM.  I could tune this some more to find where the cliff is, but this is a one-off job so now that it's completed I don't want to spend any more time tuning it.

Is there a reason that this value couldn't be dynamically adjusted in response to actual heap usage?

I can imagine a scenario where spending too much time in GC (descending into GC hell) drops the value a little to keep from OOM, or directly measuring how much of the heap is spent on this scratch space and adjusting appropriately.


On Sat, Feb 8, 2014 at 3:40 PM, Matei Zaharia <[hidden email]> wrote:
This probably means that there’s not enough free memory for the “scratch” space used for computations, so we OOM before the Spark cache decides that it’s full and starts to spill stuff. Try reducing spark.storage.memoryFraction (default is 0.66, try 0.5).

Matei

On Feb 5, 2014, at 10:29 PM, Andrew Ash <[hidden email]> wrote:

// version 0.9.0

Hi Spark users,

My understanding of the MEMORY_AND_DISK_SER persistence level was that if an RDD could fit into memory then it would be left there (same as MEMORY_ONLY), and only if it was too big for memory would it spill to disk.  Here's how the docs describe it:

MEMORY_AND_DISK_SER Similar to MEMORY_ONLY_SER, but spill partitions that don't fit in memory to disk instead of recomputing them on the fly each time they're needed.



What I'm observing though is that really large RDDs are actually causing OOMs.  I'm not sure if this is a regression in 0.9.0 or if it has been this way for some time.

While I look through the source code, has anyone actually observed the correct spill to disk behavior rather than an OOM?

Thanks!
Andrew