Quantcast

Standalone Spark connecting to kerberos enabled hadoop cluster

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Standalone Spark connecting to kerberos enabled hadoop cluster

preethi.krishnan
This post has NOT been accepted by the mailing list yet.
Hi,

I'm trying to write files to kerberos enabled hadoop cluster from a standalone spark streaming listening to kafka. It fails to connect to hadoop with the following exception

Exception in thread "main" java.io.IOException: Login failure for <user>@XXX.XX-XX.COM from keytab /home/batch/kerberos/dev7/.<user>.keytab
        at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytabAndReturnUGI(UserGroupInformation.java:1117)


Caused by: java.lang.IllegalArgumentException: Illegal principal name <user>@XXX.XX-XX.COM: org.apache.hadoop.security.authentication.util.KerberosName$NoMatchingRule: No rules applied to <user>@XX-XX.WAL-MART.COM
        at org.apache.hadoop.security.User.<init>(User.java:50)
        at org.apache.hadoop.security.User.<init>(User.java:43)
        at org.apache.hadoop.security.UserGroupInformation$HadoopLoginModule.commit(UserGroupInformation.java:185)
        ... 29 more
Caused by: org.apache.hadoop.security.authentication.util.KerberosName$NoMatchingRule: No rules applied to <user>@XXX.XX-XX.COM
        at org.apache.hadoop.security.authentication.util.KerberosName.getShortName(KerberosName.java:389)

It fails to connect when I use spark-submit. However , if I use a standalone java or scala class the connection is successful.

Did anyone experienced this kind of issue?

Loading...