PySpark Streaming : Accessing the Remote Secured Kafka

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

PySpark Streaming : Accessing the Remote Secured Kafka

Ramaswamy, Muthuraman

All,

 

Currently, I am using PySpark Streaming (Classic Regular DStream Style and not Structured Streaming). Now, our remote Kafka is secured with Kerberos.

 

To enable PySpark Streaming to access the secured Kafka, what steps I should perform? Can I pass the principal/keytab and jaas config in the Spark Submit command? What are my options? Any pointers/links to access the secured Kafka broker from PySpark Streaming will be helpful.

 

Appreciate your time.

 

Thank you,

 

~Muthu