Running Spark on Mesos

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Running Spark on Mesos

deric
This post has NOT been accepted by the mailing list yet.
Hi,

I'm trying to run Spark on Mesos 0.16-rc2 and I'm getting this exception:

14/01/13 15:55:30 ERROR Utils: Unexpected NOT to have spark.hostPort set
java.lang.Exception
        at org.apache.spark.util.Utils$.logErrorWithStack(Utils.scala:442)
        at org.apache.spark.SparkEnv$.create(SparkEnv.scala:139)
        at org.apache.spark.executor.Executor.<init>(Executor.scala:110)
        at org.apache.spark.executor.MesosExecutorBackend.registered(MesosExecutorBackend.scala:58)


Where can I set this `spark.hostPort`?

Thanks,
Tomas
Reply | Threaded
Open this post in threaded view
|

Re: Running Spark on Mesos

deric
I've updated to the newest trunk version, and still all tasks are getting lost:

java.lang.NullPointerException
        at com.typesafe.config.impl.Parseable$ParseableResources.rawParseValue(Parseable.java:509)
        at com.typesafe.config.impl.Parseable$ParseableResources.rawParseValue(Parseable.java:492)
        at com.typesafe.config.impl.Parseable.parseValue(Parseable.java:171)
        at com.typesafe.config.impl.Parseable.parseValue(Parseable.java:165)
        at com.typesafe.config.impl.Parseable.parse(Parseable.java:204)
        at com.typesafe.config.ConfigFactory.parseResources(ConfigFactory.java:760)
        at com.typesafe.config.ConfigFactory.parseResources(ConfigFactory.java:769)
        at org.apache.spark.SparkConf.<init>(SparkConf.scala:37)
        at org.apache.spark.executor.Executor.<init>(Executor.scala:60)
        at org.apache.spark.executor.MesosExecutorBackend.registered(MesosExecutorBackend.scala:58)
Exception in thread "Thread-0"
Reply | Threaded
Open this post in threaded view
|

Re: Running Spark on Mesos

Benjamin Hindman
What version of Mesos are you using?

We tagged a release-candidate of Mesos that had a bug when used with frameworks that were using older JARs (like Spark). The manifestation of the bug was some protocol buffers didn't parse, such as SlaveInfo, resulting in a NullPointerException.

Until Spark gets a new JAR (and assuming you're not using the buggy release candidate of Mesos) then you can fix this problem by restarting your slave after removing it's meta-directory.

Please share the version (in fact, versions of Mesos that you've upgraded over time would be great). And let us know how it goes!

Ben.




On Mon, Jan 13, 2014 at 10:19 AM, deric <[hidden email]> wrote:
I've updated to the newest trunk version, and still all tasks are getting
lost:

java.lang.NullPointerException
        at
com.typesafe.config.impl.Parseable$ParseableResources.rawParseValue(Parseable.java:509)
        at
com.typesafe.config.impl.Parseable$ParseableResources.rawParseValue(Parseable.java:492)
        at com.typesafe.config.impl.Parseable.parseValue(Parseable.java:171)
        at com.typesafe.config.impl.Parseable.parseValue(Parseable.java:165)
        at com.typesafe.config.impl.Parseable.parse(Parseable.java:204)
        at com.typesafe.config.ConfigFactory.parseResources(ConfigFactory.java:760)
        at com.typesafe.config.ConfigFactory.parseResources(ConfigFactory.java:769)
        at org.apache.spark.SparkConf.<init>(SparkConf.scala:37)
        at org.apache.spark.executor.Executor.<init>(Executor.scala:60)
        at
org.apache.spark.executor.MesosExecutorBackend.registered(MesosExecutorBackend.scala:58)
Exception in thread "Thread-0"



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Running-Spark-on-Mesos-tp503p505.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

Reply | Threaded
Open this post in threaded view
|

Re: Running Spark on Mesos

deric
Hi Ben, 

is it possible that I've checked out that buggy version from 0.16.0-rc2 branch? Before that I was running on 0.16.0~c0a3fcf (some version from November). Which Mesos version would you recommend for running Spark? Spark's pom.xml says 0.13.0, which is quite old. 

Thanks,
Tomas


On 13 January 2014 22:14, Benjamin Hindman [via Apache Spark User List] <[hidden email]> wrote:
What version of Mesos are you using?

We tagged a release-candidate of Mesos that had a bug when used with frameworks that were using older JARs (like Spark). The manifestation of the bug was some protocol buffers didn't parse, such as SlaveInfo, resulting in a NullPointerException.

Until Spark gets a new JAR (and assuming you're not using the buggy release candidate of Mesos) then you can fix this problem by restarting your slave after removing it's meta-directory.

Please share the version (in fact, versions of Mesos that you've upgraded over time would be great). And let us know how it goes!

Ben.




On Mon, Jan 13, 2014 at 10:19 AM, deric <[hidden email]> wrote:
I've updated to the newest trunk version, and still all tasks are getting
lost:

java.lang.NullPointerException
        at
com.typesafe.config.impl.Parseable$ParseableResources.rawParseValue(Parseable.java:509)
        at
com.typesafe.config.impl.Parseable$ParseableResources.rawParseValue(Parseable.java:492)
        at com.typesafe.config.impl.Parseable.parseValue(Parseable.java:171)
        at com.typesafe.config.impl.Parseable.parseValue(Parseable.java:165)
        at com.typesafe.config.impl.Parseable.parse(Parseable.java:204)
        at com.typesafe.config.ConfigFactory.parseResources(ConfigFactory.java:760)
        at com.typesafe.config.ConfigFactory.parseResources(ConfigFactory.java:769)
        at org.apache.spark.SparkConf.<init>(SparkConf.scala:37)
        at org.apache.spark.executor.Executor.<init>(Executor.scala:60)
        at
org.apache.spark.executor.MesosExecutorBackend.registered(MesosExecutorBackend.scala:58)
Exception in thread "Thread-0"



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Running-Spark-on-Mesos-tp503p505.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.




If you reply to this email, your message will be added to the discussion below:
http://apache-spark-user-list.1001560.n3.nabble.com/Running-Spark-on-Mesos-tp503p516.html
To unsubscribe from Running Spark on Mesos, click here.
NAML

Reply | Threaded
Open this post in threaded view
|

Re: Running Spark on Mesos

Benjamin Hindman
You should be able to use 0.16.0-rc2, but I recommend using 0.16.0-rc3 since it fixes a bug with the webui (but not related to this).

Did you try restarting your slaves after deleting the meta-directory? Kill the slave, delete the meta-directory (by default at /tmp/mesos/meta unless you passed --work_dir to the slave) and then restart the slave. If things don't work after that please let us know!

Ben.


On Mon, Jan 13, 2014 at 4:06 PM, deric <[hidden email]> wrote:
Hi Ben, 

is it possible that I've checked out that buggy version from 0.16.0-rc2 branch? Before that I was running on 0.16.0~c0a3fcf (some version from November). Which Mesos version would you recommend for running Spark? Spark's pom.xml says 0.13.0, which is quite old. 

Thanks,
Tomas


On 13 January 2014 22:14, Benjamin Hindman [via Apache Spark User List] <[hidden email]> wrote:
What version of Mesos are you using?

We tagged a release-candidate of Mesos that had a bug when used with frameworks that were using older JARs (like Spark). The manifestation of the bug was some protocol buffers didn't parse, such as SlaveInfo, resulting in a NullPointerException.

Until Spark gets a new JAR (and assuming you're not using the buggy release candidate of Mesos) then you can fix this problem by restarting your slave after removing it's meta-directory.

Please share the version (in fact, versions of Mesos that you've upgraded over time would be great). And let us know how it goes!

Ben.




On Mon, Jan 13, 2014 at 10:19 AM, deric <[hidden email]> wrote:
I've updated to the newest trunk version, and still all tasks are getting
lost:

java.lang.NullPointerException
        at
com.typesafe.config.impl.Parseable$ParseableResources.rawParseValue(Parseable.java:509)
        at
com.typesafe.config.impl.Parseable$ParseableResources.rawParseValue(Parseable.java:492)
        at com.typesafe.config.impl.Parseable.parseValue(Parseable.java:171)
        at com.typesafe.config.impl.Parseable.parseValue(Parseable.java:165)
        at com.typesafe.config.impl.Parseable.parse(Parseable.java:204)
        at com.typesafe.config.ConfigFactory.parseResources(ConfigFactory.java:760)
        at com.typesafe.config.ConfigFactory.parseResources(ConfigFactory.java:769)
        at org.apache.spark.SparkConf.<init>(SparkConf.scala:37)
        at org.apache.spark.executor.Executor.<init>(Executor.scala:60)
        at
org.apache.spark.executor.MesosExecutorBackend.registered(MesosExecutorBackend.scala:58)
Exception in thread "Thread-0"



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Running-Spark-on-Mesos-tp503p505.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.




If you reply to this email, your message will be added to the discussion below:
http://apache-spark-user-list.1001560.n3.nabble.com/Running-Spark-on-Mesos-tp503p516.html
To unsubscribe from Running Spark on Mesos, click here.
NAML



View this message in context: Re: Running Spark on Mesos

Sent from the Apache Spark User List mailing list archive at Nabble.com.

Reply | Threaded
Open this post in threaded view
|

Re: Running Spark on Mesos

deric
I've deleted whole /tmp/mesos on each slave, but it didn't help (this one was running on mesos 0.15.0).  I've tried different mesos versions (0.14, 0.15, 0.16-rc1, 0.16-rc2). Now spark is compiled with mesos-0.15.0.jar, but it doesn't seem to have any impact on this.
java.lang.NullPointerException
	at com.typesafe.config.impl.Parseable$ParseableResources.rawParseValue(Parseable.java:509)
	at com.typesafe.config.impl.Parseable$ParseableResources.rawParseValue(Parseable.java:492)
	at com.typesafe.config.impl.Parseable.parseValue(Parseable.java:171)
	at com.typesafe.config.impl.Parseable.parseValue(Parseable.java:165)
	at com.typesafe.config.impl.Parseable.parse(Parseable.java:204)
	at com.typesafe.config.ConfigFactory.parseResources(ConfigFactory.java:760)
	at com.typesafe.config.ConfigFactory.parseResources(ConfigFactory.java:769)
	at org.apache.spark.SparkConf.<init>(SparkConf.scala:37)
	at org.apache.spark.executor.Executor.<init>(Executor.scala:60)
	at org.apache.spark.executor.MesosExecutorBackend.registered(MesosExecutorBackend.scala:58)
Exception in thread "Thread-0" 

stdout:
Running spark-executor with framework dir = /usr/share/spark
14/01/14 09:54:46 ERROR MesosExecutorBackend: Received launchTask but executor was null



On 14 January 2014 03:23, Benjamin Hindman [via Apache Spark User List] <[hidden email]> wrote:
You should be able to use 0.16.0-rc2, but I recommend using 0.16.0-rc3 since it fixes a bug with the webui (but not related to this).

Did you try restarting your slaves after deleting the meta-directory? Kill the slave, delete the meta-directory (by default at /tmp/mesos/meta unless you passed --work_dir to the slave) and then restart the slave. If things don't work after that please let us know!

Ben.


On Mon, Jan 13, 2014 at 4:06 PM, deric <[hidden email]> wrote:
Hi Ben, 

is it possible that I've checked out that buggy version from 0.16.0-rc2 branch? Before that I was running on 0.16.0~c0a3fcf (some version from November). Which Mesos version would you recommend for running Spark? Spark's pom.xml says 0.13.0, which is quite old. 

Thanks,
Tomas


On 13 January 2014 22:14, Benjamin Hindman [via Apache Spark User List] <[hidden email]> wrote:
What version of Mesos are you using?

We tagged a release-candidate of Mesos that had a bug when used with frameworks that were using older JARs (like Spark). The manifestation of the bug was some protocol buffers didn't parse, such as SlaveInfo, resulting in a NullPointerException.

Until Spark gets a new JAR (and assuming you're not using the buggy release candidate of Mesos) then you can fix this problem by restarting your slave after removing it's meta-directory.

Please share the version (in fact, versions of Mesos that you've upgraded over time would be great). And let us know how it goes!

Ben.




On Mon, Jan 13, 2014 at 10:19 AM, deric <[hidden email]> wrote:
I've updated to the newest trunk version, and still all tasks are getting
lost:

java.lang.NullPointerException
        at
com.typesafe.config.impl.Parseable$ParseableResources.rawParseValue(Parseable.java:509)
        at
com.typesafe.config.impl.Parseable$ParseableResources.rawParseValue(Parseable.java:492)
        at com.typesafe.config.impl.Parseable.parseValue(Parseable.java:171)
        at com.typesafe.config.impl.Parseable.parseValue(Parseable.java:165)
        at com.typesafe.config.impl.Parseable.parse(Parseable.java:204)
        at com.typesafe.config.ConfigFactory.parseResources(ConfigFactory.java:760)
        at com.typesafe.config.ConfigFactory.parseResources(ConfigFactory.java:769)
        at org.apache.spark.SparkConf.<init>(SparkConf.scala:37)
        at org.apache.spark.executor.Executor.<init>(Executor.scala:60)
        at
org.apache.spark.executor.MesosExecutorBackend.registered(MesosExecutorBackend.scala:58)
Exception in thread "Thread-0"



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Running-Spark-on-Mesos-tp503p505.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.




If you reply to this email, your message will be added to the discussion below:
http://apache-spark-user-list.1001560.n3.nabble.com/Running-Spark-on-Mesos-tp503p516.html
To unsubscribe from Running Spark on Mesos, click here.
NAML



View this message in context: Re: Running Spark on Mesos

Sent from the Apache Spark User List mailing list archive at Nabble.com.




If you reply to this email, your message will be added to the discussion below:
http://apache-spark-user-list.1001560.n3.nabble.com/Running-Spark-on-Mesos-tp503p528.html
To unsubscribe from Running Spark on Mesos, click here.
NAML