Spark Streaming to REST API

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Spark Streaming to REST API

ashish rawat
Hi,

We are working on a streaming solution where multiple out of order streams are flowing in the system and we need to join the streams based on a unique id. We are planning to use redis for this, where for every tuple, we will lookup if the id exists, we join if it does or else put the tuple into redis. Also, we need to write the final out to a system through REST API (the system doesn't provide any other mechanism to write). 

Is it a common pattern to read/write to db per tuple? Also, are there any connectors to write to REST endpoints.

Regards,
Ashish
Reply | Threaded
Open this post in threaded view
|

Re: Spark Streaming to REST API

Gourav Sengupta
hi Ashish,

I was just wondering if there is any particular reason why you are posting this to a SPARK group?

Regards,
Gourav

On Thu, Dec 21, 2017 at 8:32 PM, ashish rawat <[hidden email]> wrote:
Hi,

We are working on a streaming solution where multiple out of order streams are flowing in the system and we need to join the streams based on a unique id. We are planning to use redis for this, where for every tuple, we will lookup if the id exists, we join if it does or else put the tuple into redis. Also, we need to write the final out to a system through REST API (the system doesn't provide any other mechanism to write). 

Is it a common pattern to read/write to db per tuple? Also, are there any connectors to write to REST endpoints.

Regards,
Ashish

Reply | Threaded
Open this post in threaded view
|

Re: Spark Streaming to REST API

ashish rawat
Sorry, for not making it explicit. We are using Spark Streaming as the streaming solution and I was wondering if it is a common pattern to do per tuple redis read/write and write to a REST API through Spark Streaming.

Regards,
Ashish

On Fri, Dec 22, 2017 at 4:00 AM, Gourav Sengupta <[hidden email]> wrote:
hi Ashish,

I was just wondering if there is any particular reason why you are posting this to a SPARK group?

Regards,
Gourav

On Thu, Dec 21, 2017 at 8:32 PM, ashish rawat <[hidden email]> wrote:
Hi,

We are working on a streaming solution where multiple out of order streams are flowing in the system and we need to join the streams based on a unique id. We are planning to use redis for this, where for every tuple, we will lookup if the id exists, we join if it does or else put the tuple into redis. Also, we need to write the final out to a system through REST API (the system doesn't provide any other mechanism to write). 

Is it a common pattern to read/write to db per tuple? Also, are there any connectors to write to REST endpoints.

Regards,
Ashish