Pyspark - Kerberos authentication error

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Pyspark - Kerberos authentication error

Pravinkumar vp
Hello, I’m new to pyspark programming and facing Issue when try to connect the HDFS with Kerberos auth principal and keytab.

My environment is a docker container environment, so installed necessary libraries including pyspark, krb5-* for client

In my scenario, before creating spark context I have done the kinit to get the token. After that I’m creating spark context and trying to connect but it’s failing with the following error.

Org.apache.hadoop.security.AccessControlException: SIMPLE authentication is not enabled. Available: [TOKEN, KERBEROS]

Please let me know what is the issue here.. and also please share some sample code to create a spark session to connect the Kerberos enabled cluster with principal and keytab

Thanks!!