Status of Spark testing on ARM64

[huangtianhua]

Hi all,

I will give you some informations about ARM CI of Spark:

Our team and community are working on build/test Spark master on ARM64 server, after find and fix some issues[1], we have integrated two ARM testing jobs[2] to community CI(AMPLAB Jenkins),

they run as daily job and have been stablely running for few weeks, and the two ARM testing jobs are success generally. 

Thanks Sean Owen, Shane Knapp, Dongjoon Hyun and community to help us :)

If you are interested, please have a try:)  Before https://github.com/apache/spark/pull/26636 merged, you have to download and maven install org.openlabtesting.leveldbjni:leveldbjni-all.1.8 using commands:

wget https://repo1.maven.org/maven2/org/openlabtesting/leveldbjni/leveldbjni-all/1.8/leveldbjni-all-1.8.jar 

mvn install:install-file -DgroupId=org.fusesource.leveldbjni -DartifactId=leveldbjni-all -Dversion=1.8 -Dpackaging=jar -Dfile=leveldbjni-all-1.8.jar

Then, you can build and test Spark on ARM64 server.

 

If you have any questions, please don't hesitate to contact me, thanks all!

[1]:

https://issues.apache.org/jira/browse/SPARK-28770 （https://github.com/apache/spark/pull/25673）
https://issues.apache.org/jira/browse/SPARK-28519 （https://github.com/apache/spark/pull/25279）
https://issues.apache.org/jira/browse/SPARK-28433 （https://github.com/apache/spark/pull/25186）
https://issues.apache.org/jira/browse/SPARK-28467 （https://github.com/apache/spark/pull/25864）
https://issues.apache.org/jira/browse/SPARK-29286 （https://github.com/apache/spark/pull/26021）

https://issues.apache.org/jira/browse/SPARK-29286   (https://github.com/apache/spark/pull/26636) --- this one is in progress

[2]:  

   https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-maven-arm/

     The job spark-master-test-maven-arm same with community x86 job https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-maven-hadoop-2.7/,

     It runs all java/scala tests, the test number is about 21,112.

   https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-python-arm/

     The job spark-master-test-python-arm runs pyspark tests with python3.6.