Spark Structured Streaming not connecting to Kafka using kerberos

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Spark Structured Streaming not connecting to Kafka using kerberos

Darshan Pandya
Hello, 

I'm using Spark 2.1.0 on CDH 5.8 with kafka 0.10.0.1 + kerberos

I am unable to connect to the kafka broker with the following message 


17/10/14 14:29:10 WARN clients.NetworkClient: Bootstrap broker 10.197.19.25:9092 disconnected

and is unable to consume any messages.

And am using it as follows 

jaas.conf

KafkaClient {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
keyTab="./gandalf.keytab"
storeKey=true
useTicketCache=false
serviceName="kafka"
principal="[hidden email]";
};
$SPARK_HOME/bin/spark-submit \
--master yarn \
--files jaas.conf,gandalf.keytab \
--driver-java-options "-Djava.security.auth.login.config=./jaas.conf -Dhdp.version=2.4.2.0-258" \
--conf "spark.executor.extraJavaOptions=-Djava.security.auth.login.config=./jaas.conf" \
--class com.example.ClassName uber-jar-with-deps-and-hive-site.jar
Thanks in advance. 

--
Sincerely, 
Darshan

Reply | Threaded
Open this post in threaded view
|

Unsubscribe

serkan ta?


Android için Outlook uygulamasını edinin


From: Darshan Pandya <[hidden email]>
Sent: Saturday, October 14, 2017 11:55:06 AM
To: user
Subject: Spark Structured Streaming not connecting to Kafka using kerberos
 
Hello, 

I'm using Spark 2.1.0 on CDH 5.8 with kafka 0.10.0.1 + kerberos

I am unable to connect to the kafka broker with the following message 


17/10/14 14:29:10 WARN clients.NetworkClient: Bootstrap broker 10.197.19.25:9092 disconnected

and is unable to consume any messages.

And am using it as follows 

jaas.conf

KafkaClient {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
keyTab="./gandalf.keytab"
storeKey=true
useTicketCache=false
serviceName="kafka"
principal="[hidden email]";
};
$SPARK_HOME/bin/spark-submit \
--master yarn \
--files jaas.conf,gandalf.keytab \
--driver-java-options "-Djava.security.auth.login.config=./jaas.conf -Dhdp.version=2.4.2.0-258" \
--conf "spark.executor.extraJavaOptions=-Djava.security.auth.login.config=./jaas.conf" \
--class com.example.ClassName uber-jar-with-deps-and-hive-site.jar
Thanks in advance. 

--
Sincerely, 
Darshan

Reply | Threaded
Open this post in threaded view
|

Re: Spark Structured Streaming not connecting to Kafka using kerberos

Burak Yavuz-2
In reply to this post by Darshan Pandya
Hi Darshan,

How are you creating your kafka stream? Can you please share the options you provide?

spark.readStream.format("kafka")
  .option(...) // all these please
  .load()


On Sat, Oct 14, 2017 at 1:55 AM, Darshan Pandya <[hidden email]> wrote:
Hello, 

I'm using Spark 2.1.0 on CDH 5.8 with kafka 0.10.0.1 + kerberos

I am unable to connect to the kafka broker with the following message 


17/10/14 14:29:10 WARN clients.NetworkClient: Bootstrap broker 10.197.19.25:9092 disconnected

and is unable to consume any messages.

And am using it as follows 

jaas.conf

KafkaClient {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
keyTab="./gandalf.keytab"
storeKey=true
useTicketCache=false
serviceName="kafka"
principal="[hidden email]";
};
$SPARK_HOME/bin/spark-submit \
--master yarn \
--files jaas.conf,gandalf.keytab \
--driver-java-options "-Djava.security.auth.login.config=./jaas.conf -Dhdp.version=2.4.2.0-258" \
--conf "spark.executor.extraJavaOptions=-Djava.security.auth.login.config=./jaas.conf" \
--class com.example.ClassName uber-jar-with-deps-and-hive-site.jar
Thanks in advance. 

--
Sincerely, 
Darshan


Reply | Threaded
Open this post in threaded view
|

Re: Spark Structured Streaming not connecting to Kafka using kerberos

Darshan Pandya
HI Burak, 

Well turns out it worked fine when i submit in cluster mode. I also tried to convert my app in dstreams. In dstreams too it works well only when deployed in cluster mode. 

Here is how i configured the stream.


val lines = spark.readStream
.format("kafka")
.option("kafka.bootstrap.servers", jobParams.boorstrapServer)
.option("subscribe", jobParams.sourceTopic)
.option("startingOffsets", "latest")
.option("minPartitions", "10")
.option("failOnDataLoss", "true")
.load()


Sincerely, 
Darshan 

On Mon, Oct 16, 2017 at 12:08 PM, Burak Yavuz <[hidden email]> wrote:
Hi Darshan,

How are you creating your kafka stream? Can you please share the options you provide?

spark.readStream.format("kafka")
  .option(...) // all these please
  .load()


On Sat, Oct 14, 2017 at 1:55 AM, Darshan Pandya <[hidden email]> wrote:
Hello, 

I'm using Spark 2.1.0 on CDH 5.8 with kafka 0.10.0.1 + kerberos

I am unable to connect to the kafka broker with the following message 


17/10/14 14:29:10 WARN clients.NetworkClient: Bootstrap broker 10.197.19.25:9092 disconnected

and is unable to consume any messages.

And am using it as follows 

jaas.conf

KafkaClient {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
keyTab="./gandalf.keytab"
storeKey=true
useTicketCache=false
serviceName="kafka"
principal="[hidden email]";
};
$SPARK_HOME/bin/spark-submit \
--master yarn \
--files jaas.conf,gandalf.keytab \
--driver-java-options "-Djava.security.auth.login.config=./jaas.conf -Dhdp.version=2.4.2.0-258" \
--conf "spark.executor.extraJavaOptions=-Djava.security.auth.login.config=./jaas.conf" \
--class com.example.ClassName uber-jar-with-deps-and-hive-site.jar
Thanks in advance. 

--
Sincerely, 
Darshan





--
Sincerely, 
Darshan

Reply | Threaded
Open this post in threaded view
|

Re: Spark Structured Streaming not connecting to Kafka using kerberos

Prashant Sharma
Hi Darshan,

Did you try passing the config directly as an option, like this: 
.option("kafka.sasl.jaas.config", saslConfig)

Where saslConfig can look like: 
com.sun.security.auth.module.Krb5LoginModule required \
        useKeyTab=true \
        storeKey=true  \
        keyTab="/etc/security/keytabs/kafka_client.keytab" \
        principal="[hidden email]";

Thanks,
Prashant.

On Tue, Oct 17, 2017 at 11:21 AM, Darshan Pandya <[hidden email]> wrote:
HI Burak, 

Well turns out it worked fine when i submit in cluster mode. I also tried to convert my app in dstreams. In dstreams too it works well only when deployed in cluster mode. 

Here is how i configured the stream.


val lines = spark.readStream
.format("kafka")
.option("kafka.bootstrap.servers", jobParams.boorstrapServer)
.option("subscribe", jobParams.sourceTopic)
.option("startingOffsets", "latest")
.option("minPartitions", "10")
.option("failOnDataLoss", "true")
.load()


Sincerely, 
Darshan 

On Mon, Oct 16, 2017 at 12:08 PM, Burak Yavuz <[hidden email]> wrote:
Hi Darshan,

How are you creating your kafka stream? Can you please share the options you provide?

spark.readStream.format("kafka")
  .option(...) // all these please
  .load()


On Sat, Oct 14, 2017 at 1:55 AM, Darshan Pandya <[hidden email]> wrote:
Hello, 

I'm using Spark 2.1.0 on CDH 5.8 with kafka 0.10.0.1 + kerberos

I am unable to connect to the kafka broker with the following message 


17/10/14 14:29:10 WARN clients.NetworkClient: Bootstrap broker 10.197.19.25:9092 disconnected

and is unable to consume any messages.

And am using it as follows 

jaas.conf

KafkaClient {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
keyTab="./gandalf.keytab"
storeKey=true
useTicketCache=false
serviceName="kafka"
principal="[hidden email]";
};
$SPARK_HOME/bin/spark-submit \
--master yarn \
--files jaas.conf,gandalf.keytab \
--driver-java-options "-Djava.security.auth.login.config=./jaas.conf -Dhdp.version=2.4.2.0-258" \
--conf "spark.executor.extraJavaOptions=-Djava.security.auth.login.config=./jaas.conf" \
--class com.example.ClassName uber-jar-with-deps-and-hive-site.jar
Thanks in advance. 

--
Sincerely, 
Darshan





--
Sincerely, 
Darshan