configuring .sparkStaging with group rwx

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

configuring .sparkStaging with group rwx

Bulldog20630405

we have a spark cluster running on with multiple users...
when running with the user owning the cluster jobs run fine... however, when trying to run pyspark with a different user it fails because the .sparkStaging/application_* is written with 700 so the user cannot write to that directory....

how to configure spark/yarn cluster so .sparkStaging is written as 770 instead of 700 so the users with shared group can execute?


Reply | Threaded
Open this post in threaded view
|

Re: configuring .sparkStaging with group rwx

"Yuri Oleynikov (‫יורי אולייניקוב‬‎)"
Spark-submit --conf spark.hadoop.fs.permissions.umask-mode=007
You may also set sticky bit on staging dir 

Sent from my iPhone

On 26 Feb 2021, at 03:29, Bulldog20630405 <[hidden email]> wrote:



we have a spark cluster running on with multiple users...
when running with the user owning the cluster jobs run fine... however, when trying to run pyspark with a different user it fails because the .sparkStaging/application_* is written with 700 so the user cannot write to that directory....

how to configure spark/yarn cluster so .sparkStaging is written as 770 instead of 700 so the users with shared group can execute?