list of persisted rdds

classic Classic list List threaded Threaded
7 messages Options
list of persisted rdds – Hi, How do I check the rdds that I have persisted? I have some code that looks like: "rd1.cache() .... rd2.cache() ... .... rdN.cach...
Check out SparkContext.getPersistentRDDs! On Fri, Jun 13, 2014 at 1:06 PM, mrm <maria@...> wrote: > Hi, > > How do I chec...
Hi Daniel, Thank you for your help! This is the sort of thing I was looking for. However, when I type "sc.getPersistentRDDs", i get t...
val myRdds = sc.getPersistentRDDs assert(myRdds.size === 1) It'll return a map. Its pretty old 0.8.0 onwards. Regards Mayur ...
This appears to be missing from PySpark. Reported in SPARK-2141 <https://issues.apache.org/jira/browse/SPARK-2141>. On Fri, Jun 13, ...
Hi Nick, Thank you for the reply, I forgot to mention I was using pyspark in my first message. Maria
Yeah, unfortunately PySpark still lags behind the Scala API a bit, but it's being patched up at a good pace. On Fri, Jun 13, 2014 at 1:43 PM...