use spark cluster in java web service

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

use spark cluster in java web service

崔苗(数据与人工智能产品开发部)
Hi,
we want to use spark in our java web service , compute data in spark cluster according to request,now we have two probles:
1、 how to get sparkSession of remote spark cluster (spark on yarn mode) , we want to keep one sparkSession to execute all data compution;
2、how to submit to remote spark cluster in java code instead of spark-submit , as we want to execute spark code in reponse server;

Thanks for any replys
--------------------------------------------------------------------- To unsubscribe e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

Fwd: use spark cluster in java web service

onmstester onmstester-2
Refer:
1. Create a singleton SparkContext at initialization of your cluster, the spark-context or spark-sql would be accessible through a static method anywhere in your application.
I recommend using Fair scheduling on your context, to share resources among all input requests
    SparkSession spark = SparkSession.builder().appName("Simple Application").getOrCreate();

2. From now on, with sc or spark-sql object, something like sparkSql.sql("select * from test").collectAsList() would be run as a spark job and returns result to your application

Sent using Zoho Mail



============ Forwarded message ============
From : 崔苗(数据与人工智能产品开发部) <[hidden email]>
To : "user"<[hidden email]>
Date : Thu, 01 Nov 2018 10:52:15 +0330
Subject : use spark cluster in java web service
============ Forwarded message ============


Hi,
we want to use spark in our java web service , compute data in spark cluster according to request,now we have two probles:
1、 how to get sparkSession of remote spark cluster (spark on yarn mode) , we want to keep one sparkSession to execute all data compution;
2、how to submit to remote spark cluster in java code instead of spark-submit , as we want to execute spark code in reponse server;

Thanks for any replys

--------------------------------------------------------------------- To unsubscribe e-mail:[hidden email]


Reply | Threaded
Open this post in threaded view
|

Re: use spark cluster in java web service

hemant singh
In reply to this post by 崔苗(数据与人工智能产品开发部)
Why do't you explore Livy. You can use the Rest API to submit the jobs - https://community.hortonworks.com/articles/151164/how-to-submit-spark-application-through-livy-rest.html



On Thu, Nov 1, 2018 at 12:52 PM 崔苗(数据与人工智能产品开发部) <[hidden email]> wrote:
Hi,
we want to use spark in our java web service , compute data in spark cluster according to request,now we have two probles:
1、 how to get sparkSession of remote spark cluster (spark on yarn mode) , we want to keep one sparkSession to execute all data compution;
2、how to submit to remote spark cluster in java code instead of spark-submit , as we want to execute spark code in reponse server;

Thanks for any replys