Spark 2.4.4 with Hadoop 3.2.0

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Spark 2.4.4 with Hadoop 3.2.0

bsikander
Hi,
Are Spark 2.4.4 and Hadoop 3.2.0 compatible?
I tried to search the mailing list but couldn't find anything relevant.

 



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Spark 2.4.4 with Hadoop 3.2.0

Alfredo Marquez
I also would like know the answer to this question.

Thanks,

Alfredo

On Tue, Nov 19, 2019, 8:24 AM bsikander <[hidden email]> wrote:
Hi,
Are Spark 2.4.4 and Hadoop 3.2.0 compatible?
I tried to search the mailing list but couldn't find anything relevant.





--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Spark 2.4.4 with Hadoop 3.2.0

Ismaël Mejía
Not officially. Apache Spark only announced support for Hadoop 3.x starting with the upcoming Spark 3.
There is a preview release of Spark 3 with support for Hadoop 3.2 that you can try now:

Enjoy!



On Tue, Nov 19, 2019 at 3:44 PM Alfredo Marquez <[hidden email]> wrote:
I also would like know the answer to this question.

Thanks,

Alfredo

On Tue, Nov 19, 2019, 8:24 AM bsikander <[hidden email]> wrote:
Hi,
Are Spark 2.4.4 and Hadoop 3.2.0 compatible?
I tried to search the mailing list but couldn't find anything relevant.





--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Spark 2.4.4 with Hadoop 3.2.0

Alfredo Marquez
Thank you Ismael! That's what I was looking for. I can take this to our platform team.

Alfredo

On Mon, Nov 25, 2019, 3:32 PM Ismaël Mejía <[hidden email]> wrote:
Not officially. Apache Spark only announced support for Hadoop 3.x starting with the upcoming Spark 3.
There is a preview release of Spark 3 with support for Hadoop 3.2 that you can try now:

Enjoy!



On Tue, Nov 19, 2019 at 3:44 PM Alfredo Marquez <[hidden email]> wrote:
I also would like know the answer to this question.

Thanks,

Alfredo

On Tue, Nov 19, 2019, 8:24 AM bsikander <[hidden email]> wrote:
Hi,
Are Spark 2.4.4 and Hadoop 3.2.0 compatible?
I tried to search the mailing list but couldn't find anything relevant.





--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Spark 2.4.4 with Hadoop 3.2.0

nihed mbarek
Hi, 
Spark 2.x is already part of Cloudera CDH6 who is based on Hadoop 3.x so they support officially Spark2+Hadoop3 
So for sure, there is tests and development done from this side. In other part,  I don't know the status for Hadoop 3.2.

Regards, 

On Tue, Nov 26, 2019 at 1:46 AM Alfredo Marquez <[hidden email]> wrote:
Thank you Ismael! That's what I was looking for. I can take this to our platform team.

Alfredo

On Mon, Nov 25, 2019, 3:32 PM Ismaël Mejía <[hidden email]> wrote:
Not officially. Apache Spark only announced support for Hadoop 3.x starting with the upcoming Spark 3.
There is a preview release of Spark 3 with support for Hadoop 3.2 that you can try now:

Enjoy!



On Tue, Nov 19, 2019 at 3:44 PM Alfredo Marquez <[hidden email]> wrote:
I also would like know the answer to this question.

Thanks,

Alfredo

On Tue, Nov 19, 2019, 8:24 AM bsikander <[hidden email]> wrote:
Hi,
Are Spark 2.4.4 and Hadoop 3.2.0 compatible?
I tried to search the mailing list but couldn't find anything relevant.





--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]



--

M'BAREK Med Nihed,
Fedora Ambassador, TUNISIA, Northern Africa
http://www.nihed.com



Reply | Threaded
Open this post in threaded view
|

Re: Spark 2.4.4 with Hadoop 3.2.0

bsikander
It could be that CDH6 has the integration but somehow I am getting the
following very frequently while building Spark2.4.4 with Hadoop 3.2.0 and
running spark tests.

caused by: java.lang.IllegalArgumentException: Unrecognized Hadoop major
version number: 3.2.0
    at
org.apache.hadoop.hive.shims.ShimLoader.getMajorVersion(ShimLoader.java:174)
    at
org.apache.hadoop.hive.shims.ShimLoader.loadShims(ShimLoader.java:139)
    at
org.apache.hadoop.hive.shims.ShimLoader.getHadoopShims(ShimLoader.java:100)
    at
org.apache.hadoop.hive.conf.HiveConf$ConfVars.<clinit>(HiveConf.java:368)





--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]