Cached data not showing up in Storage tab

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Cached data not showing up in Storage tab

V0lleyBallJunki3
When I cache a variable the data never shows up in the
storage tab. The storage tab is always blank. I have tried it in
Zeppelin as well as spark-shell.

scala> val classCount = spark.read.parquet("s3:// ..../classCount")
scala> classCount.persist
scala> classCount.count

Nothing shows up in the Storage tab of either Zeppelin or spark-shell.
However, I have several running applications in production that does
show the data in cache. I am using Scala and Spark 2.2.1 in EMR. Any
workarounds to see the data in cache.

The problem is mentioned here:
https://forums.databricks.com/questions/117/why-is-my-rdd-not-showing-up-in-the-storage-tab-of.html
https://stackoverflow.com/questions/44792213/blank-storage-tab-in-spark-history-server

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]