PySpark Direct Streaming : SASL Security Compatibility Issue

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

PySpark Direct Streaming : SASL Security Compatibility Issue

Ramaswamy, Muthuraman

Hi All,

 

I am using PySpark Direct Streaming to connect to a remote secured Kafka broker and is secured with Kerberos Authentication. The KafkaUtils.createDirectStream python call gives me the following error:

 

18/11/27 18:20:05 WARN VerifiableProperties: Property sasl.mechanism is not valid

18/11/27 18:20:05 WARN VerifiableProperties: Property security.protocol is not valid

 

Does spark-streaming-kafka-0-8_2.11 library support SASL_PLAINTEXT security protocol? Is there any compatibility issue with 0.8 version of the package?

 

Appreciate your help to resolve the security issue.

 

Thank you,

 

Muthu R

 

Reply | Threaded
Open this post in threaded view
|

Re: PySpark Direct Streaming : SASL Security Compatibility Issue

Gabor Somogyi
Hi Muthuraman,

Previous to 0.9, Kafka had no built-in security features but DStreams support only kafka 0.8.
Suggest to use structured streaming where 0.10+ (2.0.0 in spark 2.4) support is available.

BR,
G


On Wed, Nov 28, 2018 at 4:15 AM Ramaswamy, Muthuraman <[hidden email]> wrote:

Hi All,

 

I am using PySpark Direct Streaming to connect to a remote secured Kafka broker and is secured with Kerberos Authentication. The KafkaUtils.createDirectStream python call gives me the following error:

 

18/11/27 18:20:05 WARN VerifiableProperties: Property sasl.mechanism is not valid

18/11/27 18:20:05 WARN VerifiableProperties: Property security.protocol is not valid

 

Does spark-streaming-kafka-0-8_2.11 library support SASL_PLAINTEXT security protocol? Is there any compatibility issue with 0.8 version of the package?

 

Appreciate your help to resolve the security issue.

 

Thank you,

 

Muthu R