Spark Streaming. Real-time save data and visualize on dashboard

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Spark Streaming. Real-time save data and visualize on dashboard

tencas
This post has NOT been accepted by the mailing list yet.
I've developed an application using Apache Spark Streaming, that reads simple info from plane sensors like acceleration, via TCP sockets on json format, and analyse it.

I'd like to be able to persist this info from each "flight" on real-time, while it is shown on any responsive dashboard.

I just don't know if is it possible to use a no-SQL database like Mongo, Cassandra in collaboration with a monitoring tools like Grafana,Kabana, and it make sense.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Spark Streaming. Real-time save data and visualize on dashboard

Gaurav1809
This post has NOT been accepted by the mailing list yet.
May be you can injest your data in ELK and use Kibana for live reporting. Of course there can be better way of doing this. Waiting for others to share their opinion. Thanks.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Spark Streaming. Real-time save data and visualize on dashboard

tencas
This post has NOT been accepted by the mailing list yet.
Hi Gaurav1809 ,

I was thinking about using elasticsearch + kibana too (actually don't know the differences between ELK and elasticsearch).
I was wondering about pros and cons of using a document indexer vs NoSQL database.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Spark Streaming. Real-time save data and visualize on dashboard

Balaji Krishnan
This post has NOT been accepted by the mailing list yet.
In reply to this post by tencas
Hi..
  How about ingesting that into Kafka Topic and using Apache Presto as Query engine you can use any Visualization tool (Oracle Data Viz, Amazon Quick Sight etc) to build a near real-time and interactive dashboard.

Regards

Bala
Loading...