Is there a better way to read kerberized impala tables by spark jdbc?

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Is there a better way to read kerberized impala tables by spark jdbc?

eabour@163.com
Hi:

I want to use spark jdbc to read kerberized impala tables, like:
```
val impalaUrl = "jdbc:impala://<host_imapal_deamon>:21050;AuthMech=1;KrbRealm=REALM.COM;KrbHostFQDN=<host_impala_deamon>;KrbServiceName=impala"
spark.read.jdbc(impalaUrl)
```

As we know, spark will read impala data by executor rather than driver, so throw excepting:  javax.security.sasl.SaslException: GSS initiate failed

```
Caused by: org.ietf.jgss.GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
        at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
        at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122)
        at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
        at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224)
        at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
        at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
        at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192)
        ... 20 common frames omitted

```

Ony way to solve this problem is set jaas.conf by "java.security.auth.login.config" property, 

This is jaas.conf:

```
Client {
      com.sun.security.auth.module.Krb5LoginModule required
      useKeyTab=true
      doNotPrompt=true
      useTicketCache=true
      principal="test"
      keyTab="/home/keytab/user.keytab";
   };

```

Then set spark.executor.extraJavaOptions like :
```
--conf "spark.executor.extraJavaOptions=-Djava.security.auth.login.config=/data/disk1/spark-jdbc-impala/conf/jaas.conf -Djavax.security.auth.useSubjectCredsOnly=false" 
```

This way required absolute jaas.conf file and keyTab file, in other words, these files must be placed in the same path and on each node, Is there a better way?

Please help.

Regards



Reply | Threaded
Open this post in threaded view
|

Re: Is there a better way to read kerberized impala tables by spark jdbc?

Gabor Somogyi
At the moment I can't think of any better but we've added custom JdbcConnectionProvider API in Spark 3.1.
Hope that will make life easier in the future...

G


On Tue, Dec 8, 2020 at 3:55 AM [hidden email] <[hidden email]> wrote:
Hi:

I want to use spark jdbc to read kerberized impala tables, like:
```
val impalaUrl = "jdbc:impala://<host_imapal_deamon>:21050;AuthMech=1;KrbRealm=REALM.COM;KrbHostFQDN=<host_impala_deamon>;KrbServiceName=impala"
spark.read.jdbc(impalaUrl)
```

As we know, spark will read impala data by executor rather than driver, so throw excepting:  javax.security.sasl.SaslException: GSS initiate failed

```
Caused by: org.ietf.jgss.GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
        at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
        at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122)
        at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
        at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224)
        at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
        at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
        at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192)
        ... 20 common frames omitted

```

Ony way to solve this problem is set jaas.conf by "java.security.auth.login.config" property, 

This is jaas.conf:

```
Client {
      com.sun.security.auth.module.Krb5LoginModule required
      useKeyTab=true
      doNotPrompt=true
      useTicketCache=true
      principal="test"
      keyTab="/home/keytab/user.keytab";
   };

```

Then set spark.executor.extraJavaOptions like :
```
--conf "spark.executor.extraJavaOptions=-Djava.security.auth.login.config=/data/disk1/spark-jdbc-impala/conf/jaas.conf -Djavax.security.auth.useSubjectCredsOnly=false" 
```

This way required absolute jaas.conf file and keyTab file, in other words, these files must be placed in the same path and on each node, Is there a better way?

Please help.

Regards