a question about "take"

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

a question about "take"

Chen Jin

I just started to fiddle with Spark a week ago. I am trying to understand take function more. I have a JavaRDD<T> variable, let’s say dataSet which is initialized by getSequenceFileAsRDD, when I try dataSet.take(1), it throughs an error as follows:

java.lang.IllegalStateException: Must not use direct buffers with

But when I add add the following lines before the take call: 

        dataSet.context().setCheckpointDir("/tmp/checkpoint/", true);


Then take() works. I am confused here by how RDD's take method. Can anybody help me explain why this happens?

Thanks a lot,