Is there a step-by-step instruction on how to build Spark App with IntelliJ IDEA?

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Is there a step-by-step instruction on how to build Spark App with IntelliJ IDEA?

Wei Da
Hi guys,
I'm using IntelliJ IDEA 13.1.2 Community Edition, and I have installed Scala plugin and Maven 3.2.1. I want to develop Spark applications with IntelliJ IDEA through Maven. 

In IntelliJ, I create a Maven project with the archetype ID "spark-core_2.10", but got the following messages in the "Message Maven Goal":

=================================================================================================================================

[WARNING] Archetype not found in any catalog. Falling back to central repository (http://repo1.maven.org/maven2).
[WARNING] Use -DarchetypeRepository=<your repository> if archetype's repository is elsewhere.
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 20.064 s
[INFO] Finished at: 2014-06-02T11:50:14+08:00
[INFO] Final Memory: 9M/65M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-archetype-plugin:2.2:generate (default-cli) on project standalone-pom: The defined artifact is not an archetype -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] Maven execution terminated abnormally (exit code 1)

=================================================================================================================================

I have spent several days on this, but did not get any success. 
The intructions on Spark Website (http://spark.apache.org/docs/latest/building-with-maven.html) may be to brief for newbies like me. Is there any more detailed instructions on how to build Spark App with Intellij IDEA? Thanks a lot!
Reply | Threaded
Open this post in threaded view
|

Re: Is there a step-by-step instruction on how to build Spark App with IntelliJ IDEA?

Matei Zaharia
Administrator
Don’t try to use spark-core as an archetype. Instead just create a plain Scala project (no archetype) and add a Maven dependency on spark-core. That should be all you need.

Matei

On Jun 1, 2014, at 9:15 PM, Wei Da <[hidden email]> wrote:

Hi guys,
I'm using IntelliJ IDEA 13.1.2 Community Edition, and I have installed Scala plugin and Maven 3.2.1. I want to develop Spark applications with IntelliJ IDEA through Maven. 

In IntelliJ, I create a Maven project with the archetype ID "spark-core_2.10", but got the following messages in the "Message Maven Goal":

=================================================================================================================================

[WARNING] Archetype not found in any catalog. Falling back to central repository (http://repo1.maven.org/maven2).
[WARNING] Use -DarchetypeRepository=<your repository> if archetype's repository is elsewhere.
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 20.064 s
[INFO] Finished at: 2014-06-02T11:50:14+08:00
[INFO] Final Memory: 9M/65M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-archetype-plugin:2.2:generate (default-cli) on project standalone-pom: The defined artifact is not an archetype -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] Maven execution terminated abnormally (exit code 1)

=================================================================================================================================

I have spent several days on this, but did not get any success. 
The intructions on Spark Website (http://spark.apache.org/docs/latest/building-with-maven.html) may be to brief for newbies like me. Is there any more detailed instructions on how to build Spark App with Intellij IDEA? Thanks a lot!