detect idle sparkcontext to release resources

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

detect idle sparkcontext to release resources

Nicolas Paris-2
hi

we have many users on the spark on yarn cluster. most of them forget to
release their sparkcontext after analysis (spark-thrift or pyspark
jupyter kernels).

I wonder how to detect their is no activity on the sparkcontext to kill
them.

Thanks
--
nicolas

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]