spark-submit parameters about two keytab files to yarn and kafka

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

spark-submit parameters about two keytab files to yarn and kafka

big data
Hi,

We want to submit spark streaming job to YARN and consume Kafka topic.

YARN and Kafka are in two different clusters, and they have the
different kerberos authentication.

We have two keytab files for YARN and Kafka.

And my questions is how to add parameter for spark-submit command for
this situation?

Thanks.


---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: spark-submit parameters about two keytab files to yarn and kafka

Gabor Somogyi
Hi,

Cross-realm trust must be configured. One can find several docs on how to do that.

BR,
G


On Wed, Oct 28, 2020 at 8:21 AM big data <[hidden email]> wrote:
Hi,

We want to submit spark streaming job to YARN and consume Kafka topic.

YARN and Kafka are in two different clusters, and they have the
different kerberos authentication.

We have two keytab files for YARN and Kafka.

And my questions is how to add parameter for spark-submit command for
this situation?

Thanks.


---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: spark-submit parameters about two keytab files to yarn and kafka

Kevin Pis
Hi,

Hope it can solve the issue by following method:

step 1 : 
create a kafka kerberos config named kafka_client_jaas.conf:

KafkaClient {
   com.sun.security.auth.module.Krb5LoginModule required
   useKeyTab=true
   keyTab="./kafka.service.keytab"
   storeKey=true
   useTicketCache=false
   serviceName="kafka"
   principal="kafka/[hidden email]";
};


step 2:
spark-submit command :

/usr/local/spark/bin/spark-submit \
--files ./kafka_client_jaas.conf,./kafka.service.keytab \
--driver-java-options "-Djava.security.auth.login.config=./kafka_client_jaas.conf" \
--conf "spark.executor.extraJavaOptions=-Djava.security.auth.login.config=./kafka_client_jaas.conf" \
--conf spark.yarn.keytab=./hadoop.service.keytab \
--conf spark.yarn.principal=hadoop/EXAMPLE.COM \

.....

step 3:

change security.protocol in kafka client config  to SASL_PLAINTEXT, if your spark version is 1.6.


note:
my test env:  spark 2.0.2  kafka 0.10

references  



-- 

Best,

Kevin Pis


Gabor Somogyi <[hidden email]> 于2020年10月28日周三 下午5:25写道:
Hi,

Cross-realm trust must be configured. One can find several docs on how to do that.

BR,
G


On Wed, Oct 28, 2020 at 8:21 AM big data <[hidden email]> wrote:
Hi,

We want to submit spark streaming job to YARN and consume Kafka topic.

YARN and Kafka are in two different clusters, and they have the
different kerberos authentication.

We have two keytab files for YARN and Kafka.

And my questions is how to add parameter for spark-submit command for
this situation?

Thanks.


---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]



---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

kafka_client_jaas.conf (338 bytes) Download Attachment