RESTful Operations

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

RESTful Operations

hamishberridge
I am new to Spark. The task I want to accomplish is let client send http requests, then spark process that request for further operations. However  searching Spark's website docs



I do not find any places mentioning about this. Also most of the internet result are more lated to spark job server. 

Any places I should start if I want to use Spark for such purpose?

Thanks
Reply | Threaded
Open this post in threaded view
|

Re: RESTful Operations

Chris Teoh
Maybe something like Livy, otherwise roll your own REST API and have it start a Spark job.

On Mon, 20 Jan 2020 at 06:55, <[hidden email]> wrote:
I am new to Spark. The task I want to accomplish is let client send http requests, then spark process that request for further operations. However  searching Spark's website docs



I do not find any places mentioning about this. Also most of the internet result are more lated to spark job server. 

Any places I should start if I want to use Spark for such purpose?

Thanks


--
Chris
Reply | Threaded
Open this post in threaded view
|

Re: RESTful Operations

hamishberridge
In reply to this post by hamishberridge
Sorry didn't explain well. Livy seems to me similar to job server by interacting through restful api interface for  job submission. The task I want to achieve is more like general restful api interface where user would provide some parameters for business operations (a bit like readText or a custom date source with embedded http). So when the spark receive those requests in e.g custom date source, the spark job can then perform normal operations such as  aggregation, window functions, and so on.

Thank you for the suggestions and help.

   


Jan 20, 2020, 00:26 by [hidden email]:
Maybe something like Livy, otherwise roll your own REST API and have it start a Spark job.

On Mon, 20 Jan 2020 at 06:55, <[hidden email]> wrote:
I am new to Spark. The task I want to accomplish is let client send http requests, then spark process that request for further operations. However  searching Spark's website docs



I do not find any places mentioning about this. Also most of the internet result are more lated to spark job server. 

Any places I should start if I want to use Spark for such purpose?

Thanks


--
Chris

Reply | Threaded
Open this post in threaded view
|

Re: RESTful Operations

hemant singh
Livy has both statement based(scala) as well as batch processing(code jar packaged). I think first statement based approach is what you might want to look at.
Data has to residing in some source.

Thanks,
Hemant 

On Mon, 20 Jan 2020 at 2:04 PM, <[hidden email]> wrote:
Sorry didn't explain well. Livy seems to me similar to job server by interacting through restful api interface for  job submission. The task I want to achieve is more like general restful api interface where user would provide some parameters for business operations (a bit like readText or a custom date source with embedded http). So when the spark receive those requests in e.g custom date source, the spark job can then perform normal operations such as  aggregation, window functions, and so on.

Thank you for the suggestions and help.

   


Jan 20, 2020, 00:26 by [hidden email]:
Maybe something like Livy, otherwise roll your own REST API and have it start a Spark job.

On Mon, 20 Jan 2020 at 06:55, <[hidden email]> wrote:
I am new to Spark. The task I want to accomplish is let client send http requests, then spark process that request for further operations. However  searching Spark's website docs



I do not find any places mentioning about this. Also most of the internet result are more lated to spark job server. 

Any places I should start if I want to use Spark for such purpose?

Thanks


--
Chris

Reply | Threaded
Open this post in threaded view
|

Re: RESTful Operations

hamishberridge
I see. Thanks for the advice!


Jan 20, 2020, 09:47 by [hidden email]:
Livy has both statement based(scala) as well as batch processing(code jar packaged). I think first statement based approach is what you might want to look at.
Data has to residing in some source.

Thanks,
Hemant 

On Mon, 20 Jan 2020 at 2:04 PM, <[hidden email]> wrote:
Sorry didn't explain well. Livy seems to me similar to job server by interacting through restful api interface for  job submission. The task I want to achieve is more like general restful api interface where user would provide some parameters for business operations (a bit like readText or a custom date source with embedded http). So when the spark receive those requests in e.g custom date source, the spark job can then perform normal operations such as  aggregation, window functions, and so on.

Thank you for the suggestions and help.

   


Jan 20, 2020, 00:26 by [hidden email]:

Maybe something like Livy, otherwise roll your own REST API and have it start a Spark job.

On Mon, 20 Jan 2020 at 06:55, <[hidden email]> wrote:
I am new to Spark. The task I want to accomplish is let client send http requests, then spark process that request for further operations. However  searching Spark's website docs



I do not find any places mentioning about this. Also most of the internet result are more lated to spark job server. 

Any places I should start if I want to use Spark for such purpose?

Thanks


--
Chris