how to use cluster sparkSession like localSession

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

how to use cluster sparkSession like localSession

崔苗(数据与人工智能产品开发部)

Hi,
we want to execute spark code with out submit application.jar,like this code:

public static void main(String args[]) throws Exception{
        SparkSession spark = SparkSession
                .builder()
                .master("local[*]")
                .appName("spark test")
                .getOrCreate();
      
        Dataset<Row> testData = spark.read().csv(".\\src\\main\\java\\Resources\\no_schema_iris.scv");
        testData.printSchema();
        testData.show();
    }

the above code can work well with idea , do not need to generate jar file and submit , but if we replace master("local[*]") with master("yarn") , it can't work , so is there a way to use cluster sparkSession like local sparkSession ?  we need to dynamically execute spark code in web server according to the different request ,  such as filter request will call dataset.filter() , so there is no application.jar to submit .
 
--------------------------------------------------------------------- To unsubscribe e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: how to use cluster sparkSession like localSession

Daniel de Oliveira Mantovani
Please, read about Spark Streaming or Spark Structured Streaming. Your web application can easily communicate through some API and you won’t have the overhead of start a new spark job, which is pretty heavy.

On Thu, Nov 1, 2018 at 23:01 崔苗(数据与人工智能产品开发部) <[hidden email]> wrote:

Hi,
we want to execute spark code with out submit application.jar,like this code:

public static void main(String args[]) throws Exception{
        SparkSession spark = SparkSession
                .builder()
                .master("local[*]")
                .appName("spark test")
                .getOrCreate();
      
        Dataset<Row> testData = spark.read().csv(".\\src\\main\\java\\Resources\\no_schema_iris.scv");
        testData.printSchema();
        testData.show();
    }

the above code can work well with idea , do not need to generate jar file and submit , but if we replace master("local[*]") with master("yarn") , it can't work , so is there a way to use cluster sparkSession like local sparkSession ?  we need to dynamically execute spark code in web server according to the different request ,  such as filter request will call dataset.filter() , so there is no application.jar to submit .
 
--------------------------------------------------------------------- To unsubscribe e-mail: [hidden email]
--

--
Daniel de Oliveira Mantovani
Perl Evangelist/Data Hacker
+1 786 459 1341
Reply | Threaded
Open this post in threaded view
|

Re: how to use cluster sparkSession like localSession

崔苗(数据与人工智能产品开发部)

then how about spark sql and spark MLlib , we use them at most time
On 11/2/2018 11:58[hidden email] wrote:
Please, read about Spark Streaming or Spark Structured Streaming. Your web application can easily communicate through some API and you won’t have the overhead of start a new spark job, which is pretty heavy.

On Thu, Nov 1, 2018 at 23:01 崔苗(数据与人工智能产品开发部) <[hidden email]> wrote:

Hi,
we want to execute spark code with out submit application.jar,like this code:

public static void main(String args[]) throws Exception{
        SparkSession spark = SparkSession
                .builder()
                .master("local[*]")
                .appName("spark test")
                .getOrCreate();
      
        Dataset<Row> testData = spark.read().csv(".\\src\\main\\java\\Resources\\no_schema_iris.scv");
        testData.printSchema();
        testData.show();
    }

the above code can work well with idea , do not need to generate jar file and submit , but if we replace master("local[*]") with master("yarn") , it can't work , so is there a way to use cluster sparkSession like local sparkSession ?  we need to dynamically execute spark code in web server according to the different request ,  such as filter request will call dataset.filter() , so there is no application.jar to submit .
 
--------------------------------------------------------------------- To unsubscribe e-mail: [hidden email]
--

--
Daniel de Oliveira Mantovani
Perl Evangelist/Data Hacker
+1 786 459 1341
--------------------------------------------------------------------- To unsubscribe e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: how to use cluster sparkSession like localSession

Arbab Khalil
remove master configuration from code and then submit it to any cluster, it should work.

On Fri, Nov 2, 2018 at 10:52 AM 崔苗(数据与人工智能产品开发部) <[hidden email]> wrote:

then how about spark sql and spark MLlib , we use them at most time
On 11/2/2018 11:58[hidden email] wrote:
Please, read about Spark Streaming or Spark Structured Streaming. Your web application can easily communicate through some API and you won’t have the overhead of start a new spark job, which is pretty heavy.

On Thu, Nov 1, 2018 at 23:01 崔苗(数据与人工智能产品开发部) <[hidden email]> wrote:

Hi,
we want to execute spark code with out submit application.jar,like this code:

public static void main(String args[]) throws Exception{
        SparkSession spark = SparkSession
                .builder()
                .master("local[*]")
                .appName("spark test")
                .getOrCreate();
      
        Dataset<Row> testData = spark.read().csv(".\\src\\main\\java\\Resources\\no_schema_iris.scv");
        testData.printSchema();
        testData.show();
    }

the above code can work well with idea , do not need to generate jar file and submit , but if we replace master("local[*]") with master("yarn") , it can't work , so is there a way to use cluster sparkSession like local sparkSession ?  we need to dynamically execute spark code in web server according to the different request ,  such as filter request will call dataset.filter() , so there is no application.jar to submit .
 
--------------------------------------------------------------------- To unsubscribe e-mail: [hidden email]
--

--
Daniel de Oliveira Mantovani
Perl Evangelist/Data Hacker
+1 786 459 1341


--
Regards,
Arbab Khalil
Software Design Engineer
Reply | Threaded
Open this post in threaded view
|

Re: how to use cluster sparkSession like localSession

崔苗(数据与人工智能产品开发部)
we use spark in web server , no application.jar and submit on the cluster

On 11/2/2018 13:56[hidden email] wrote:
remove master configuration from code and then submit it to any cluster, it should work.

On Fri, Nov 2, 2018 at 10:52 AM 崔苗(数据与人工智能产品开发部) <[hidden email]> wrote:

then how about spark sql and spark MLlib , we use them at most time
On 11/2/2018 11:58[hidden email] wrote:
Please, read about Spark Streaming or Spark Structured Streaming. Your web application can easily communicate through some API and you won’t have the overhead of start a new spark job, which is pretty heavy.

On Thu, Nov 1, 2018 at 23:01 崔苗(数据与人工智能产品开发部) <[hidden email]> wrote:

Hi,
we want to execute spark code with out submit application.jar,like this code:

public static void main(String args[]) throws Exception{
        SparkSession spark = SparkSession
                .builder()
                .master("local[*]")
                .appName("spark test")
                .getOrCreate();
      
        Dataset<Row> testData = spark.read().csv(".\\src\\main\\java\\Resources\\no_schema_iris.scv");
        testData.printSchema();
        testData.show();
    }

the above code can work well with idea , do not need to generate jar file and submit , but if we replace master("local[*]") with master("yarn") , it can't work , so is there a way to use cluster sparkSession like local sparkSession ?  we need to dynamically execute spark code in web server according to the different request ,  such as filter request will call dataset.filter() , so there is no application.jar to submit .
 
--------------------------------------------------------------------- To unsubscribe e-mail: [hidden email]
--

--
Daniel de Oliveira Mantovani
Perl Evangelist/Data Hacker
+1 786 459 1341


--
Regards,
Arbab Khalil
Software Design Engineer
--------------------------------------------------------------------- To unsubscribe e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: how to use cluster sparkSession like localSession

KevinZwx
In reply to this post by 崔苗(数据与人工智能产品开发部)
I think you should investigate apache zeppelin and livy
崔苗(数据与人工智能产品开发部) <[hidden email]>于2018年11月2日 周五11:01写道:

Hi,
we want to execute spark code with out submit application.jar,like this code:

public static void main(String args[]) throws Exception{
        SparkSession spark = SparkSession
                .builder()
                .master("local[*]")
                .appName("spark test")
                .getOrCreate();
      
        Dataset<Row> testData = spark.read().csv(".\\src\\main\\java\\Resources\\no_schema_iris.scv");
        testData.printSchema();
        testData.show();
    }

the above code can work well with idea , do not need to generate jar file and submit , but if we replace master("local[*]") with master("yarn") , it can't work , so is there a way to use cluster sparkSession like local sparkSession ?  we need to dynamically execute spark code in web server according to the different request ,  such as filter request will call dataset.filter() , so there is no application.jar to submit .
 
--------------------------------------------------------------------- To unsubscribe e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: how to use cluster sparkSession like localSession

Gabriel Wang
Agree. Spark is not designed for embedding in business applications(those traditional J2EE) for real-time interaction.

Thanks,
Gabriel



On Fri, Nov 2, 2018 at 2:36 PM 张万新 <[hidden email]> wrote:
I think you should investigate apache zeppelin and livy
崔苗(数据与人工智能产品开发部) <[hidden email]>于2018年11月2日 周五11:01写道:

Hi,
we want to execute spark code with out submit application.jar,like this code:

public static void main(String args[]) throws Exception{
        SparkSession spark = SparkSession
                .builder()
                .master("local[*]")
                .appName("spark test")
                .getOrCreate();
      
        Dataset<Row> testData = spark.read().csv(".\\src\\main\\java\\Resources\\no_schema_iris.scv");
        testData.printSchema();
        testData.show();
    }

the above code can work well with idea , do not need to generate jar file and submit , but if we replace master("local[*]") with master("yarn") , it can't work , so is there a way to use cluster sparkSession like local sparkSession ?  we need to dynamically execute spark code in web server according to the different request ,  such as filter request will call dataset.filter() , so there is no application.jar to submit .
 
--------------------------------------------------------------------- To unsubscribe e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

RE: how to use cluster sparkSession like localSession

Sun, Keith-2
In reply to this post by KevinZwx

Hello,

 

I think you can try with  below , the reason is only yarn-cllient mode is supported for your scenario.  

 

master("yarn-client")

 

 

 

Thanks very much.

Keith

From: 张万新 <[hidden email]>
Sent: Thursday, November 1, 2018 11:36 PM
To:
崔苗(数据与人工智能产品开发部) <[hidden email]>
Cc: user <[hidden email]>
Subject: Re: how to use cluster sparkSession like localSession

 

I think you should investigate apache zeppelin and livy

崔苗(数据与人工智能产品开发部) <[hidden email]>2018112 周五11:01写道:

 

Hi

we want to execute spark code with out submit application.jar,like this code:

 

public static void main(String args[]) throws Exception{

        SparkSession spark = SparkSession

                .builder()

                .master("local[*]")

                .appName("spark test")

                .getOrCreate();

      

        Dataset<Row> testData = spark.read().csv(".\\src\\main\\java\\Resources\\no_schema_iris.scv");

        testData.printSchema();

        testData.show();

    }

 

the above code can work well with idea , do not need to generate jar file and submit , but if we replace master("local[*]") with master("yarn") , it can't work , so is there a way to use cluster sparkSession like local sparkSession ?  we need to dynamically execute spark code in web server according to the different request ,  such as filter request will call dataset.filter() , so there is no application.jar to submit .
 

0049003208

签名由 网易邮箱大师 定制

--------------------------------------------------------------------- To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: how to use cluster sparkSession like localSession

Sumedh Wale
In reply to this post by 崔苗(数据与人工智能产品开发部)
Hi,

I think what you need is to have a long running Spark cluster to which you can submit jobs dynamically.

For SQL, you can start Spark's HiveServer2: https://spark.apache.org/docs/latest/sql-programming-guide.html#distributed-sql-engine
This will start a long running Spark cluster with a fixed configuration (executors, cores etc) and allows Spark to act more like a regular database. Then you can create jdbc:hive2:// JDBC connections from your app and run SQL queries/DDLs.

For other components (or even SQL), you can start a Spark jobserver: https://github.com/spark-jobserver/spark-jobserver
This will again start a long running Spark cluster. It also allows you create new SparkContexts on-the-fly though that should not be done from a web app rather configured separately by admin if required. It will require you to implement your job as a SparkJob/SparkSessionJob that will be provided pre-created SparkContext/SparkSession, and these take parameters that can be read dynamically in your implementation. You register your classes in jars separately before-hand. Then you can call those methods using REST API from your application providing it the required parameters like a remote procedure call.

Or you can try SnappyData that provides both of these (and much more) out of the box.

Regards,
Sumedh Wale
SnappyData (http://www.snappydata.io)
Documentation Download

On 02/11/18 11:22, 崔苗(数据与人工智能产品开发部) wrote:

then how about spark sql and spark MLlib , we use them at most time
On 11/2/2018 11:58[hidden email] wrote:
Please, read about Spark Streaming or Spark Structured Streaming. Your web application can easily communicate through some API and you won’t have the overhead of start a new spark job, which is pretty heavy.

On Thu, Nov 1, 2018 at 23:01 崔苗(数据与人工智能产品开发部) <[hidden email]> wrote:

Hi,
we want to execute spark code with out submit application.jar,like this code:

public static void main(String args[]) throws Exception{
        SparkSession spark = SparkSession
                .builder()
                .master("local[*]")
                .appName("spark test")
                .getOrCreate();
      
        Dataset<Row> testData = spark.read().csv(".\\src\\main\\java\\Resources\\no_schema_iris.scv");
        testData.printSchema();
        testData.show();
    }

the above code can work well with idea , do not need to generate jar file and submit , but if we replace master("local[*]") with master("yarn") , it can't work , so is there a way to use cluster sparkSession like local sparkSession ?  we need to dynamically execute spark code in web server according to the different request ,  such as filter request will call dataset.filter() , so there is no application.jar to submit .
 
--------------------------------------------------------------------- To unsubscribe e-mail: [hidden email]
--

--
Daniel de Oliveira Mantovani
Perl Evangelist/Data Hacker
+1 786 459 1341
--------------------------------------------------------------------- To unsubscribe e-mail: [hidden email]

--------------------------------------------------------------------- To unsubscribe e-mail: [hidden email]