Building Spark 0.9.x for CDH5 with mrv1 installation (Protobuf 2.5 upgrade)

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Building Spark 0.9.x for CDH5 with mrv1 installation (Protobuf 2.5 upgrade)

Gary Malouf
Today, our cluster setup is as follows:

Mesos 0.15, 
CDH 4.2.1-MRV1, 
Spark 0.9-pre-scala-2.10 off master build targeted at appropriate CDH4 version


We are looking to upgrade all of these in order to get protobuf 2.5 working properly.  The question is, which 'Hadoop version build' of Spark 0.9 is compatible with the HDFS from Hadoop 2.2 and Cloudera's CDH5 MRV1 installation?  Is there one?
Reply | Threaded
Open this post in threaded view
|

Re: Building Spark 0.9.x for CDH5 with mrv1 installation (Protobuf 2.5 upgrade)

Patrick Wendell
I'm not sure exactly how your cluster is configured. But as far as I can tell Cloudera's MR1 CDH5 dependencies are against Hadoop 2.3. I'd just find the exact CDH version you have and link against the `mr1` version of their published dependencies in that version.

So I think you wan't "2.3.0-mr1-cdh5.0.0"
https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/hadoop/hadoop-client/2.3.0-mr1-cdh5.0.0/

The full list of Cloudera versions is here:
https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/hadoop/hadoop-client/



On Tue, Mar 25, 2014 at 6:42 PM, Gary Malouf <[hidden email]> wrote:
Today, our cluster setup is as follows:

Mesos 0.15, 
CDH 4.2.1-MRV1, 
Spark 0.9-pre-scala-2.10 off master build targeted at appropriate CDH4 version


We are looking to upgrade all of these in order to get protobuf 2.5 working properly.  The question is, which 'Hadoop version build' of Spark 0.9 is compatible with the HDFS from Hadoop 2.2 and Cloudera's CDH5 MRV1 installation?  Is there one?