Hi Sudhir,
> On 21 Jan 2021, at 16:24, Sudhir Babu Pothineni <
[hidden email]> wrote:
>
> Any other insights into this issue? I tried multiple way to supply keytab to executor
>
> Does spark standalone doesn’t support Kerberos?
Spark standalone mode does not support Kerberos authentication. Related source code:
https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala#L346 // Kerberos is not supported in standalone mode, and keytab support is not yet available
// in Mesos cluster mode.
if (clusterManager != STANDALONE
&& !isMesosCluster
&& args.principal != null
&& args.keytab != null) {
// If client mode, make sure the keytab is just a local path.
if (deployMode == CLIENT && Utils.isLocalUri(args.keytab)) {
args.keytab = new URI(args.keytab).getPath()
}
If you want to test your application with Kerberos, I recommend for you local mode.
https://spark.apache.org/docs/latest/submitting-applications.html#master-urlsFor example:
spark-shell --master local
and if you want to access a HDFS filesystem, then you need to add the following parameter as well: spark.kerberos.access.hadoopFileSystems (in old Spark versions this is spark.yarn.access.hadoopFileSystems)
spark-shell --master local --conf spark.kerberos.access.hadoopFileSystems=hdfs://namenode.example.com:8020
It will create for you the necessary HDFS delegation token. Here is a very good documentation about the delegation token handling in Spark
https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/security/README.mdBest regards,
Gabor
---------------------------------------------------------------------
To unsubscribe e-mail:
[hidden email]