Pyspark - Kerberos authentication error

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

Pyspark - Kerberos authentication error

Pravinkumar vp
Hello, I’m new to pyspark programming and facing Issue when try to connect the HDFS with Kerberos auth principal and keytab.

My environment is a docker container environment, so installed necessary libraries including pyspark, krb5-* for client

In my scenario, before creating spark context I have done the kinit to get the token. After that I’m creating spark context and trying to connect but it’s failing with the following error. SIMPLE authentication is not enabled. Available: [TOKEN, KERBEROS]

Please let me know what is the issue here.. and also please share some sample code to create a spark session to connect the Kerberos enabled cluster with principal and keytab