Re: Unable to connect spark 0.8.1 (built for hadoop 2.2.0) to connect to mesos 0.14.2

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: Unable to connect spark 0.8.1 (built for hadoop 2.2.0) to connect to mesos 0.14.2

Matei Zaharia
Administrator
(Replying on new Spark mailing list since the old one closed).

Are you sure Spark is finding your build of Mesos instead of the Apache one from Maven Central? Unfortunately, code compiled with different protobuf versions is not compatible, because the generated code by the protoc compiler changes (even though the protobufs themselves can be read across versions). Since the published Mesos still has protobuf 2.4, you may be getting that somehow.

If this doesn’t work, I’d suggest asking on the Mesos mailing list actually. They would have a better sense of why this happens, especially if you give them your build options.

Matei

On Jan 2, 2014, at 12:01 PM, Damien Dubé <[hidden email]> wrote:

I've tried it bulding spark with mesos 0.14.2 and I have the exact same error

Stack: [0x00007f82f5849000,0x00007f82f594a000],  sp=0x00007f82f59485d0,  free space=1021k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0x632d09]  jni_GetByteArrayElements+0x89
C  [libmesos-0.14.2.so+0x5e08b9]  mesos::FrameworkInfo construct<mesos::FrameworkInfo>(JNIEnv_*, _jobject*)+0x79

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j  org.apache.mesos.MesosSchedulerDriver.initialize()V+0
j  org.apache.mesos.MesosSchedulerDriver.<init>(Lorg/apache/mesos/Scheduler;Lorg/apache/mesos/Protos$FrameworkInfo;Ljava/lang/String;)V+62
j  org.apache.spark.scheduler.cluster.mesos.MesosSchedulerBackend$$anon$1.run()V+44
v  ~StubRoutines::call_stub


On Monday, December 30, 2013 7:18:57 PM UTC-5, Jey Kottalam wrote:
It looks like your Spark is built against mesos 0.13.0 according to the stacktrace.  You may need to rebuild Spark to link with your custom build of Mesos 0.14.2.

-Jey


On Mon, Dec 30, 2013 at 1:39 PM, Damien Dubé <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="cLEmvIASWLIJ" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">damie...@...> wrote:
Once I have my mesos cluster up and running. My spark job is always returning the same error. I have tried multiple options but I am still having the same errror.

Here is the stack trace

Stack: [0x00007f41ea4c1000,0x00007f41ea5c2000],  sp=0x00007f41ea5c0670,  free space=1021k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0x632d09]  jni_GetByteArrayElements+0x89
C  [<a href="http://libmesos-0.13.0.so/" target="_blank" onmousedown="this.href='http://www.google.com/url?q\75http%3A%2F%2Flibmesos-0.13.0.so\46sa\75D\46sntz\0751\46usg\75AFQjCNEBOEvQ_YGmYuLCkPCKEIlM0_VvUQ';return true;" onclick="this.href='http://www.google.com/url?q\75http%3A%2F%2Flibmesos-0.13.0.so\46sa\75D\46sntz\0751\46usg\75AFQjCNEBOEvQ_YGmYuLCkPCKEIlM0_VvUQ';return true;">libmesos-0.13.0.so+0x5a4559]  mesos::FrameworkInfo construct<mesos::FrameworkInfo>(JNIEnv_*, _jobject*)+0x79
C  0x00007f420c0798a8
j  org.apache.spark.scheduler.cluster.mesos.MesosSchedulerBackend$$anon$1.run()V+44
v  ~StubRoutines::call_stub
V  [libjvm.so+0x5f8485]  JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*)+0x365
V  [libjvm.so+0x5f6ee8]  JavaCalls::call(JavaValue*, methodHandle, JavaCallArguments*, Thread*)+0x28
V  [libjvm.so+0x5f71b7]  JavaCalls::call_virtual(JavaValue*, KlassHandle, Symbol*, Symbol*, JavaCallArguments*, Thread*)+0x197
V  [libjvm.so+0x5f72d7]  JavaCalls::call_virtual(JavaValue*, Handle, KlassHandle, Symbol*, Symbol*, Thread*)+0x47
V  [libjvm.so+0x6731e5]  thread_entry(JavaThread*, Thread*)+0xe5
V  [libjvm.so+0x94d38f]  JavaThread::thread_main_inner()+0xdf
V  [libjvm.so+0x94d495]  JavaThread::run()+0xf5
V  [libjvm.so+0x815288]  java_start(Thread*)+0x108


What I am trying to have

Spark 0.8.1 
Mesos 0.14.2
HDFS 2.2.0 (I do not care about yarn or hadoop mapred since I am using mesos).
Oracle Java 1.7.0-45

Here are the 4 options I have tried for spark.

SPARK_HADOOP_VERSION=2.2.0 sbt/sbt assembly
and
SPARK_HADOOP_VERSION=2.2.0 SPARK_YARN=true sbt/sbt assembly

then

make-distribution.sh --hadoop 2.2.0 --with-yarn
and 
make-distribution.sh --hadoop 2.2.0



Since all of those options are built with protobuf 2.5.0

I've rebuild mesos 0.14.2 using protobuf 2.5.0

The error I am having still seems to be related to protobuf seriously do not know how to try to debug that. All my modules are now using protobuf 2.5.0.


Any ideas?


--
You received this message because you are subscribed to the Google Groups "Spark Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="cLEmvIASWLIJ" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">spark-users...@googlegroups.com.
For more options, visit <a href="https://groups.google.com/groups/opt_out" target="_blank" onmousedown="this.href='https://groups.google.com/groups/opt_out';return true;" onclick="this.href='https://groups.google.com/groups/opt_out';return true;">https://groups.google.com/groups/opt_out.


--
You received this message because you are subscribed to the Google Groups "Spark Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.

Reply | Threaded
Open this post in threaded view
|

Re: Unable to connect spark 0.8.1 (built for hadoop 2.2.0) to connect to mesos 0.14.2

Jey Kottalam
One way to ensure that the right build of Mesos is found is to change
the version number. For example, give your build of Mesos with
Protobuf 2.5 the version number "0.14.2-pb2.5", and reference that
version in your Spark build.

-Jey

On Fri, Jan 3, 2014 at 9:14 PM, Matei Zaharia <[hidden email]> wrote:

> (Replying on new Spark mailing list since the old one closed).
>
> Are you sure Spark is finding your build of Mesos instead of the Apache one
> from Maven Central? Unfortunately, code compiled with different protobuf
> versions is not compatible, because the generated code by the protoc
> compiler changes (even though the protobufs themselves can be read across
> versions). Since the published Mesos still has protobuf 2.4, you may be
> getting that somehow.
>
> If this doesn’t work, I’d suggest asking on the Mesos mailing list actually.
> They would have a better sense of why this happens, especially if you give
> them your build options.
>
> Matei
>
>
> On Jan 2, 2014, at 12:01 PM, Damien Dubé <[hidden email]> wrote:
>
> I've tried it bulding spark with mesos 0.14.2 and I have the exact same
> error
>
> Stack: [0x00007f82f5849000,0x00007f82f594a000],  sp=0x00007f82f59485d0,
> free space=1021k
> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native
> code)
> V  [libjvm.so+0x632d09]  jni_GetByteArrayElements+0x89
> C  [libmesos-0.14.2.so+0x5e08b9]  mesos::FrameworkInfo
> construct<mesos::FrameworkInfo>(JNIEnv_*, _jobject*)+0x79
>
> Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
> j  org.apache.mesos.MesosSchedulerDriver.initialize()V+0
> j
> org.apache.mesos.MesosSchedulerDriver.<init>(Lorg/apache/mesos/Scheduler;Lorg/apache/mesos/Protos$FrameworkInfo;Ljava/lang/String;)V+62
> j
> org.apache.spark.scheduler.cluster.mesos.MesosSchedulerBackend$$anon$1.run()V+44
> v  ~StubRoutines::call_stub
>
>
> On Monday, December 30, 2013 7:18:57 PM UTC-5, Jey Kottalam wrote:
>>
>> It looks like your Spark is built against mesos 0.13.0 according to the
>> stacktrace.  You may need to rebuild Spark to link with your custom build of
>> Mesos 0.14.2.
>>
>> -Jey
>>
>>
>> On Mon, Dec 30, 2013 at 1:39 PM, Damien Dubé <[hidden email]> wrote:
>>>
>>> Once I have my mesos cluster up and running. My spark job is always
>>> returning the same error. I have tried multiple options but I am still
>>> having the same errror.
>>>
>>> Here is the stack trace
>>>
>>> Stack: [0x00007f41ea4c1000,0x00007f41ea5c2000],  sp=0x00007f41ea5c0670,
>>> free space=1021k
>>> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native
>>> code)
>>> V  [libjvm.so+0x632d09]  jni_GetByteArrayElements+0x89
>>> C  [libmesos-0.13.0.so+0x5a4559]  mesos::FrameworkInfo
>>> construct<mesos::FrameworkInfo>(JNIEnv_*, _jobject*)+0x79
>>> C  0x00007f420c0798a8
>>> j
>>> org.apache.spark.scheduler.cluster.mesos.MesosSchedulerBackend$$anon$1.run()V+44
>>> v  ~StubRoutines::call_stub
>>> V  [libjvm.so+0x5f8485]  JavaCalls::call_helper(JavaValue*,
>>> methodHandle*, JavaCallArguments*, Thread*)+0x365
>>> V  [libjvm.so+0x5f6ee8]  JavaCalls::call(JavaValue*, methodHandle,
>>> JavaCallArguments*, Thread*)+0x28
>>> V  [libjvm.so+0x5f71b7]  JavaCalls::call_virtual(JavaValue*, KlassHandle,
>>> Symbol*, Symbol*, JavaCallArguments*, Thread*)+0x197
>>> V  [libjvm.so+0x5f72d7]  JavaCalls::call_virtual(JavaValue*, Handle,
>>> KlassHandle, Symbol*, Symbol*, Thread*)+0x47
>>> V  [libjvm.so+0x6731e5]  thread_entry(JavaThread*, Thread*)+0xe5
>>> V  [libjvm.so+0x94d38f]  JavaThread::thread_main_inner()+0xdf
>>> V  [libjvm.so+0x94d495]  JavaThread::run()+0xf5
>>> V  [libjvm.so+0x815288]  java_start(Thread*)+0x108
>>>
>>>
>>> What I am trying to have
>>>
>>> Spark 0.8.1
>>> Mesos 0.14.2
>>> HDFS 2.2.0 (I do not care about yarn or hadoop mapred since I am using
>>> mesos).
>>> Oracle Java 1.7.0-45
>>>
>>> Here are the 4 options I have tried for spark.
>>>
>>> SPARK_HADOOP_VERSION=2.2.0 sbt/sbt assembly
>>> and
>>> SPARK_HADOOP_VERSION=2.2.0 SPARK_YARN=true sbt/sbt assembly
>>>
>>> then
>>>
>>> make-distribution.sh --hadoop 2.2.0 --with-yarn
>>> and
>>> make-distribution.sh --hadoop 2.2.0
>>>
>>>
>>>
>>> Since all of those options are built with protobuf 2.5.0
>>>
>>> I've rebuild mesos 0.14.2 using protobuf 2.5.0
>>>
>>> The error I am having still seems to be related to protobuf seriously do
>>> not know how to try to debug that. All my modules are now using protobuf
>>> 2.5.0.
>>>
>>>
>>> Any ideas?
>>>
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "Spark Users" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an
>>> email to [hidden email].
>>> For more options, visit https://groups.google.com/groups/opt_out.
>>
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Spark Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [hidden email].
> For more options, visit https://groups.google.com/groups/opt_out.
>
>