Create custom receiver for MQTT in spark streaming

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Create custom receiver for MQTT in spark streaming

Muhammed Favas

Hi,

 

I have a requirement to do analysis using spark on the data coming from IoT device via MQTT broker. The connectivity from my spark job is with MQTT broker where I can subscribe to specific topics. I have used the MQTTUtils library in spark to connect to the broker, but I have doubts about how the library works internally. What I noticed is, "MQTTutils.createStream" connects to the MQTT broker for a topic and it is without any authentication to broker. In that case, if I have to subscribe 100 topics in an MQTT broker, it may establish 100 connections to broker (Please let me know if it is not working the way I think.). That is not desirable in my real scenario.

 

So I have decided to create a custom receiver for MQTT broker so that I can manage the connection in my MQTT client. I have gone through the document on how to implement a custom receiver, but I did not succeed in implementing it in the proper way.

 

If somebody has hands-on on such a custom receiver, please help me to make it works.

 

Appreciate your support since it is critical for my solution

 

Regards,

Favas