Quantcast

Calling spark from a java web application.

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Calling spark from a java web application.

Jamal
This post has NOT been accepted by the mailing list yet.
Hi,

Any recommendation or tutorial on calling spark from java web application.

Current setup:
A spring java web application running on Jetty.

Requirement:
Need to run queries from the web app to spark sql.

Please point or recommend the process. I can only find standalone app example on the spark webiste.

Obliged
Jamal
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Calling spark from a java web application.

adrian

On Fri, Nov 28, 2014 at 6:59 AM, Jamal [via Apache Spark User List] <[hidden email]> wrote:
Hi,

Any recommendation or tutorial on calling spark from java web application.

Current setup:
A spring java web application running on Jetty.

Requirement:
Need to run queries from the web app to spark sql.

Please point or recommend the process. I can only find standalone app example on the spark webiste.

Obliged
Jamal


If you reply to this email, your message will be added to the discussion below:
http://apache-spark-user-list.1001560.n3.nabble.com/Calling-spark-from-a-java-web-application-tp20007.html
To start a new topic under Apache Spark User List, email [hidden email]
To unsubscribe from Apache Spark User List, click here.
NAML

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Calling spark from a java web application.

ryaminal
In reply to this post by Jamal
If you are able to use YARN in your hadoop cluster, then the following technique is pretty straightforward:
http://blog.sequenceiq.com/blog/2014/08/22/spark-submit-in-java/

We use this in our system and it's super easy to execute from our Tomcat application.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: Calling spark from a java web application.

Mohammed Guller
In reply to this post by adrian

Jamal,

I have not tried this, but can you not integrate Spark SQL with your Spring Java web app just like a standalone app? I have integrated a Scala web app (using Play) with Spark SQL and it works.

 

Mohammed

 

From: adrian [mailto:[hidden email]]
Sent: Friday, November 28, 2014 11:03 AM
To: [hidden email]
Subject: Re: Calling spark from a java web application.

 

 

On Fri, Nov 28, 2014 at 6:59 AM, Jamal [via Apache Spark User List] <[hidden email]> wrote:

Hi,

Any recommendation or tutorial on calling spark from java web application.

Current setup:
A spring java web application running on Jetty.

Requirement:
Need to run queries from the web app to spark sql.

Please point or recommend the process. I can only find standalone app example on the spark webiste.

Obliged
Jamal


If you reply to this email, your message will be added to the discussion below:

http://apache-spark-user-list.1001560.n3.nabble.com/Calling-spark-from-a-java-web-application-tp20007.html

To start a new topic under Apache Spark User List, email [hidden email]
To unsubscribe from Apache Spark User List, click here.
NAML

 

 


View this message in context: Re: Calling spark from a java web application.
Sent from the Apache Spark User List mailing list archive at Nabble.com.

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Calling spark from a java web application.

nsareen
In reply to this post by Jamal
We have a web application which talks to spark server.

This is how we have done the integration.
1) In the tomcat's classpath, add the spark distribution jar for spark code to be available at runtime ( for you it would be Jetty).
2) In the Web application project, add the spark distribution jar in the classpath ( Could be Java / Web project).
3) Setup the FAIR scheduling mode, which would help send parallel requests from web application to the spark cluster.
4) In our application startup, initialize the connection to the spark cluster. This is composed of creating the JavaSparkContext and making it available throughout the web application, in case this needs to be the only Driver Program required by the web application.
5) Using the JavaSpark Context, Create RDD's and make them available globally to the web application code.
6) invoke transformation / actions as required.

Hopefully this info is of some use..
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Calling spark from a java web application.

arul_anand_2000
This post has NOT been accepted by the mailing list yet.
Can you please let me know how the log4j properties where configured. I am trying to integrate spark with web application. when spark context gets initialized, it overrides existing log4j properites.

Regards,
arul.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Calling spark from a java web application.

Ricardo Paiva
This post has NOT been accepted by the mailing list yet.
$SPARK_HOME/conf/log4j.properties

It uses by default $SPARK_HOME/conf/log4j.properties.template

On Thu, Mar 31, 2016 at 3:28 PM, arul_anand_2000 [via Apache Spark User List] <[hidden email]> wrote:
Can you please let me know how the log4j properties where configured. I am trying to integrate spark with web application. when spark context gets initialized, it overrides existing log4j properites.

Regards,
arul.


If you reply to this email, your message will be added to the discussion below:
http://apache-spark-user-list.1001560.n3.nabble.com/Calling-spark-from-a-java-web-application-tp20007p26649.html
To start a new topic under Apache Spark User List, email [hidden email]
To unsubscribe from Apache Spark User List, click here.
NAML



--
Ricardo Paiva
Big Data / Semântica
2483-6432
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Calling spark from a java web application.

hardRocker
This post has NOT been accepted by the mailing list yet.
In reply to this post by Jamal
Hi,

can you give some idea whether you could able to do it?
I am trying to do the same thing so please can you give some feedback.

Thanks and Regards
Loading...