Status of Spark testing on ARM64

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Status of Spark testing on ARM64

huangtianhua
Hi all,
I will give you some informations about ARM CI of Spark:

Our team and community are working on build/test Spark master on ARM64 server, after find and fix some issues[1], we have integrated two ARM testing jobs[2] to community CI(AMPLAB Jenkins),
they run as daily job and have been stablely running for few weeks, and the two ARM testing jobs are success generally. 
Thanks Sean Owen, Shane Knapp, Dongjoon Hyun and community to help us :)

If you are interested, please have a try:)  Before https://github.com/apache/spark/pull/26636 merged, you have to download and maven install org.openlabtesting.leveldbjni:leveldbjni-all.1.8 using commands:
mvn install:install-file -DgroupId=org.fusesource.leveldbjni -DartifactId=leveldbjni-all -Dversion=1.8 -Dpackaging=jar -Dfile=leveldbjni-all-1.8.jar
Then, you can build and test Spark on ARM64 server.
 
If you have any questions, please don't hesitate to contact me, thanks all!

[1]:

[2]:  
     The job spark-master-test-maven-arm same with community x86 job https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-maven-hadoop-2.7/,
     It runs all java/scala tests, the test number is about 21,112.
     The job spark-master-test-python-arm runs pyspark tests with python3.6.