Block

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Block

David Thomas
What is the concept of Block and BlockManager in Spark? How is a Block related to a Partition of a RDD?
Reply | Threaded
Open this post in threaded view
|

Re: Block

Patrick Wendell
A block is an internal construct that isn't directly exposed to users.
Internally though, each partition of an RDD is mapped to one block.

- Patrick

On Mon, Mar 10, 2014 at 11:06 PM, David Thomas <[hidden email]> wrote:
> What is the concept of Block and BlockManager in Spark? How is a Block
> related to a Partition of a RDD?
Reply | Threaded
Open this post in threaded view
|

Re: Block

dachuan
In reply to this post by David Thomas

In my opinion, BlockManager manages many types of Block, RDD's partition, a.k.a. RDDBlock, is one type of them. Other types of Blocks are ShuffleBlock, IndirectBlock (if the task's return status is too large), etc.

So, BlockManager is a layer that is independent of RDD concept.

On Mar 11, 2014 2:06 AM, "David Thomas" <[hidden email]> wrote:
What is the concept of Block and BlockManager in Spark? How is a Block related to a Partition of a RDD?