compile spark 3.1.1 error

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

compile spark 3.1.1 error

jiahong li
hi, everybody, when i compile spark 3.1.1 from tag v3.1.1 ,encounter error like this:

INFO] --- scala-maven-plugin:4.3.0:compile (scala-compile-first) @ spark-core_2.12 ---
[INFO] Using incremental compilation using Mixed compile order
[INFO] Compiler bridge file: .sbt/1.0/zinc/org.scala-sbt/org.scala-sbt-compiler-bridge_2.12-1.3.1-bin_2.12.10__52.0-1.3.1_20191012T045515.jar
[INFO] compiler plugin: BasicArtifact(com.github.ghik,silencer-plugin_2.12.10,1.6.0,null)
[INFO] Compiling 560 Scala sources and 99 Java sources to git/spark/core/target/scala-2.12/classes ...
[ERROR] [Error] git/spark/core/src/main/scala/org/apache/spark/ui/HttpSecurityFilter.scala:107: type mismatch;
 found   : K where type K
 required: String
[ERROR] [Error] git/spark/core/src/main/scala/org/apache/spark/ui/HttpSecurityFilter.scala:107: value map is not a member of V
[ERROR] [Error] git/spark/core/src/main/scala/org/apache/spark/ui/HttpSecurityFilter.scala:107: missing argument list for method stripXSS in class XssSafeRequest
Unapplied methods are only converted to functions when a function type is expected.
You can make this conversion explicit by writing `stripXSS _` or `stripXSS(_)` instead of `stripXSS`.
[ERROR] [Error] git/spark/core/src/main/scala/org/apache/spark/ui/PagedTable.scala:307: value startsWith is not a member of K
[ERROR] [Error] git/spark/core/src/main/scala/org/apache/spark/util/Utils.scala:580: value toLowerCase is not a member of object org.apache.hadoop.util.StringUtils
[ERROR] 5 errors found

anybody encounter error like this?
 
Reply | Threaded
Open this post in threaded view
|

Re: compile spark 3.1.1 error

Attila Zsolt Piros
hi!

Are you compiling Spark itself?
Do you use "./build/mvn" from the project root? 
If you compiled an other version of Spark before and there the scala version was different then zinc/nailgun could cached the old classes which can cause similar troubles.
In that case this could help:

./build/zinc-0.3.15/bin/zinc -shutdown

Best Regards,
Attila

On Wed, Mar 10, 2021 at 11:27 AM jiahong li <[hidden email]> wrote:
hi, everybody, when i compile spark 3.1.1 from tag v3.1.1 ,encounter error like this:

INFO] --- scala-maven-plugin:4.3.0:compile (scala-compile-first) @ spark-core_2.12 ---
[INFO] Using incremental compilation using Mixed compile order
[INFO] Compiler bridge file: .sbt/1.0/zinc/org.scala-sbt/org.scala-sbt-compiler-bridge_2.12-1.3.1-bin_2.12.10__52.0-1.3.1_20191012T045515.jar
[INFO] compiler plugin: BasicArtifact(com.github.ghik,silencer-plugin_2.12.10,1.6.0,null)
[INFO] Compiling 560 Scala sources and 99 Java sources to git/spark/core/target/scala-2.12/classes ...
[ERROR] [Error] git/spark/core/src/main/scala/org/apache/spark/ui/HttpSecurityFilter.scala:107: type mismatch;
 found   : K where type K
 required: String
[ERROR] [Error] git/spark/core/src/main/scala/org/apache/spark/ui/HttpSecurityFilter.scala:107: value map is not a member of V
[ERROR] [Error] git/spark/core/src/main/scala/org/apache/spark/ui/HttpSecurityFilter.scala:107: missing argument list for method stripXSS in class XssSafeRequest
Unapplied methods are only converted to functions when a function type is expected.
You can make this conversion explicit by writing `stripXSS _` or `stripXSS(_)` instead of `stripXSS`.
[ERROR] [Error] git/spark/core/src/main/scala/org/apache/spark/ui/PagedTable.scala:307: value startsWith is not a member of K
[ERROR] [Error] git/spark/core/src/main/scala/org/apache/spark/util/Utils.scala:580: value toLowerCase is not a member of object org.apache.hadoop.util.StringUtils
[ERROR] 5 errors found

anybody encounter error like this?
 
Reply | Threaded
Open this post in threaded view
|

Re: compile spark 3.1.1 error

jiahong li
i use ./build/mvn to compile ,and after execute command :./build/zinc-0.3.15/bin/zinc -shutdown
and execute command like this: /dev/make-distribution.sh --name custom-spark --pip  --tgz -Phive -Phive-thriftserver -Pyarn -Dhadoop.version=2.6.0-cdh5.13.1 -DskipTests
same error appear.
and execute command: ps -ef |grep zinc, there is nothing containe zinc

Attila Zsolt Piros <[hidden email]> 于2021年3月10日周三 下午6:55写道:
hi!

Are you compiling Spark itself?
Do you use "./build/mvn" from the project root? 
If you compiled an other version of Spark before and there the scala version was different then zinc/nailgun could cached the old classes which can cause similar troubles.
In that case this could help:

./build/zinc-0.3.15/bin/zinc -shutdown

Best Regards,
Attila

On Wed, Mar 10, 2021 at 11:27 AM jiahong li <[hidden email]> wrote:
hi, everybody, when i compile spark 3.1.1 from tag v3.1.1 ,encounter error like this:

INFO] --- scala-maven-plugin:4.3.0:compile (scala-compile-first) @ spark-core_2.12 ---
[INFO] Using incremental compilation using Mixed compile order
[INFO] Compiler bridge file: .sbt/1.0/zinc/org.scala-sbt/org.scala-sbt-compiler-bridge_2.12-1.3.1-bin_2.12.10__52.0-1.3.1_20191012T045515.jar
[INFO] compiler plugin: BasicArtifact(com.github.ghik,silencer-plugin_2.12.10,1.6.0,null)
[INFO] Compiling 560 Scala sources and 99 Java sources to git/spark/core/target/scala-2.12/classes ...
[ERROR] [Error] git/spark/core/src/main/scala/org/apache/spark/ui/HttpSecurityFilter.scala:107: type mismatch;
 found   : K where type K
 required: String
[ERROR] [Error] git/spark/core/src/main/scala/org/apache/spark/ui/HttpSecurityFilter.scala:107: value map is not a member of V
[ERROR] [Error] git/spark/core/src/main/scala/org/apache/spark/ui/HttpSecurityFilter.scala:107: missing argument list for method stripXSS in class XssSafeRequest
Unapplied methods are only converted to functions when a function type is expected.
You can make this conversion explicit by writing `stripXSS _` or `stripXSS(_)` instead of `stripXSS`.
[ERROR] [Error] git/spark/core/src/main/scala/org/apache/spark/ui/PagedTable.scala:307: value startsWith is not a member of K
[ERROR] [Error] git/spark/core/src/main/scala/org/apache/spark/util/Utils.scala:580: value toLowerCase is not a member of object org.apache.hadoop.util.StringUtils
[ERROR] 5 errors found

anybody encounter error like this?
 
Reply | Threaded
Open this post in threaded view
|

Re: compile spark 3.1.1 error

Attila Zsolt Piros
I see, this must be because of hadoop version you are selecting by using "-Dhadoop.version=2.6.0-cdh5.13.1". 
Spark 3.1.1 only support hadoop-2.7 and hadoop-3.2, at least these two can be given via profiles:  -Phadoop-2.7  and -Phadoop-3.2 (the default). 


On Wed, Mar 10, 2021 at 12:26 PM jiahong li <[hidden email]> wrote:
i use ./build/mvn to compile ,and after execute command :./build/zinc-0.3.15/bin/zinc -shutdown
and execute command like this: /dev/make-distribution.sh --name custom-spark --pip  --tgz -Phive -Phive-thriftserver -Pyarn -Dhadoop.version=2.6.0-cdh5.13.1 -DskipTests
same error appear.
and execute command: ps -ef |grep zinc, there is nothing containe zinc

Attila Zsolt Piros <[hidden email]> 于2021年3月10日周三 下午6:55写道:
hi!

Are you compiling Spark itself?
Do you use "./build/mvn" from the project root? 
If you compiled an other version of Spark before and there the scala version was different then zinc/nailgun could cached the old classes which can cause similar troubles.
In that case this could help:

./build/zinc-0.3.15/bin/zinc -shutdown

Best Regards,
Attila

On Wed, Mar 10, 2021 at 11:27 AM jiahong li <[hidden email]> wrote:
hi, everybody, when i compile spark 3.1.1 from tag v3.1.1 ,encounter error like this:

INFO] --- scala-maven-plugin:4.3.0:compile (scala-compile-first) @ spark-core_2.12 ---
[INFO] Using incremental compilation using Mixed compile order
[INFO] Compiler bridge file: .sbt/1.0/zinc/org.scala-sbt/org.scala-sbt-compiler-bridge_2.12-1.3.1-bin_2.12.10__52.0-1.3.1_20191012T045515.jar
[INFO] compiler plugin: BasicArtifact(com.github.ghik,silencer-plugin_2.12.10,1.6.0,null)
[INFO] Compiling 560 Scala sources and 99 Java sources to git/spark/core/target/scala-2.12/classes ...
[ERROR] [Error] git/spark/core/src/main/scala/org/apache/spark/ui/HttpSecurityFilter.scala:107: type mismatch;
 found   : K where type K
 required: String
[ERROR] [Error] git/spark/core/src/main/scala/org/apache/spark/ui/HttpSecurityFilter.scala:107: value map is not a member of V
[ERROR] [Error] git/spark/core/src/main/scala/org/apache/spark/ui/HttpSecurityFilter.scala:107: missing argument list for method stripXSS in class XssSafeRequest
Unapplied methods are only converted to functions when a function type is expected.
You can make this conversion explicit by writing `stripXSS _` or `stripXSS(_)` instead of `stripXSS`.
[ERROR] [Error] git/spark/core/src/main/scala/org/apache/spark/ui/PagedTable.scala:307: value startsWith is not a member of K
[ERROR] [Error] git/spark/core/src/main/scala/org/apache/spark/util/Utils.scala:580: value toLowerCase is not a member of object org.apache.hadoop.util.StringUtils
[ERROR] 5 errors found

anybody encounter error like this?
 
Reply | Threaded
Open this post in threaded view
|

Re: compile spark 3.1.1 error

jiahong li
it not the cause,when i set -Phadoop-2.7 instead of -Dhadoop.version=2.6.0-cdh5.13.1, the same errors come out.

Attila Zsolt Piros <[hidden email]> 于2021年3月10日周三 下午8:56写道:
I see, this must be because of hadoop version you are selecting by using "-Dhadoop.version=2.6.0-cdh5.13.1". 
Spark 3.1.1 only support hadoop-2.7 and hadoop-3.2, at least these two can be given via profiles:  -Phadoop-2.7  and -Phadoop-3.2 (the default). 


On Wed, Mar 10, 2021 at 12:26 PM jiahong li <[hidden email]> wrote:
i use ./build/mvn to compile ,and after execute command :./build/zinc-0.3.15/bin/zinc -shutdown
and execute command like this: /dev/make-distribution.sh --name custom-spark --pip  --tgz -Phive -Phive-thriftserver -Pyarn -Dhadoop.version=2.6.0-cdh5.13.1 -DskipTests
same error appear.
and execute command: ps -ef |grep zinc, there is nothing containe zinc

Attila Zsolt Piros <[hidden email]> 于2021年3月10日周三 下午6:55写道:
hi!

Are you compiling Spark itself?
Do you use "./build/mvn" from the project root? 
If you compiled an other version of Spark before and there the scala version was different then zinc/nailgun could cached the old classes which can cause similar troubles.
In that case this could help:

./build/zinc-0.3.15/bin/zinc -shutdown

Best Regards,
Attila

On Wed, Mar 10, 2021 at 11:27 AM jiahong li <[hidden email]> wrote:
hi, everybody, when i compile spark 3.1.1 from tag v3.1.1 ,encounter error like this:

INFO] --- scala-maven-plugin:4.3.0:compile (scala-compile-first) @ spark-core_2.12 ---
[INFO] Using incremental compilation using Mixed compile order
[INFO] Compiler bridge file: .sbt/1.0/zinc/org.scala-sbt/org.scala-sbt-compiler-bridge_2.12-1.3.1-bin_2.12.10__52.0-1.3.1_20191012T045515.jar
[INFO] compiler plugin: BasicArtifact(com.github.ghik,silencer-plugin_2.12.10,1.6.0,null)
[INFO] Compiling 560 Scala sources and 99 Java sources to git/spark/core/target/scala-2.12/classes ...
[ERROR] [Error] git/spark/core/src/main/scala/org/apache/spark/ui/HttpSecurityFilter.scala:107: type mismatch;
 found   : K where type K
 required: String
[ERROR] [Error] git/spark/core/src/main/scala/org/apache/spark/ui/HttpSecurityFilter.scala:107: value map is not a member of V
[ERROR] [Error] git/spark/core/src/main/scala/org/apache/spark/ui/HttpSecurityFilter.scala:107: missing argument list for method stripXSS in class XssSafeRequest
Unapplied methods are only converted to functions when a function type is expected.
You can make this conversion explicit by writing `stripXSS _` or `stripXSS(_)` instead of `stripXSS`.
[ERROR] [Error] git/spark/core/src/main/scala/org/apache/spark/ui/PagedTable.scala:307: value startsWith is not a member of K
[ERROR] [Error] git/spark/core/src/main/scala/org/apache/spark/util/Utils.scala:580: value toLowerCase is not a member of object org.apache.hadoop.util.StringUtils
[ERROR] 5 errors found

anybody encounter error like this?
 
Reply | Threaded
Open this post in threaded view
|

Re: compile spark 3.1.1 error

jiahong li
Maybe it is my environment cause 

jiahong li <[hidden email]> 于2021年3月11日周四 上午11:14写道:
it not the cause,when i set -Phadoop-2.7 instead of -Dhadoop.version=2.6.0-cdh5.13.1, the same errors come out.

Attila Zsolt Piros <[hidden email]> 于2021年3月10日周三 下午8:56写道:
I see, this must be because of hadoop version you are selecting by using "-Dhadoop.version=2.6.0-cdh5.13.1". 
Spark 3.1.1 only support hadoop-2.7 and hadoop-3.2, at least these two can be given via profiles:  -Phadoop-2.7  and -Phadoop-3.2 (the default). 


On Wed, Mar 10, 2021 at 12:26 PM jiahong li <[hidden email]> wrote:
i use ./build/mvn to compile ,and after execute command :./build/zinc-0.3.15/bin/zinc -shutdown
and execute command like this: /dev/make-distribution.sh --name custom-spark --pip  --tgz -Phive -Phive-thriftserver -Pyarn -Dhadoop.version=2.6.0-cdh5.13.1 -DskipTests
same error appear.
and execute command: ps -ef |grep zinc, there is nothing containe zinc

Attila Zsolt Piros <[hidden email]> 于2021年3月10日周三 下午6:55写道:
hi!

Are you compiling Spark itself?
Do you use "./build/mvn" from the project root? 
If you compiled an other version of Spark before and there the scala version was different then zinc/nailgun could cached the old classes which can cause similar troubles.
In that case this could help:

./build/zinc-0.3.15/bin/zinc -shutdown

Best Regards,
Attila

On Wed, Mar 10, 2021 at 11:27 AM jiahong li <[hidden email]> wrote:
hi, everybody, when i compile spark 3.1.1 from tag v3.1.1 ,encounter error like this:

INFO] --- scala-maven-plugin:4.3.0:compile (scala-compile-first) @ spark-core_2.12 ---
[INFO] Using incremental compilation using Mixed compile order
[INFO] Compiler bridge file: .sbt/1.0/zinc/org.scala-sbt/org.scala-sbt-compiler-bridge_2.12-1.3.1-bin_2.12.10__52.0-1.3.1_20191012T045515.jar
[INFO] compiler plugin: BasicArtifact(com.github.ghik,silencer-plugin_2.12.10,1.6.0,null)
[INFO] Compiling 560 Scala sources and 99 Java sources to git/spark/core/target/scala-2.12/classes ...
[ERROR] [Error] git/spark/core/src/main/scala/org/apache/spark/ui/HttpSecurityFilter.scala:107: type mismatch;
 found   : K where type K
 required: String
[ERROR] [Error] git/spark/core/src/main/scala/org/apache/spark/ui/HttpSecurityFilter.scala:107: value map is not a member of V
[ERROR] [Error] git/spark/core/src/main/scala/org/apache/spark/ui/HttpSecurityFilter.scala:107: missing argument list for method stripXSS in class XssSafeRequest
Unapplied methods are only converted to functions when a function type is expected.
You can make this conversion explicit by writing `stripXSS _` or `stripXSS(_)` instead of `stripXSS`.
[ERROR] [Error] git/spark/core/src/main/scala/org/apache/spark/ui/PagedTable.scala:307: value startsWith is not a member of K
[ERROR] [Error] git/spark/core/src/main/scala/org/apache/spark/util/Utils.scala:580: value toLowerCase is not a member of object org.apache.hadoop.util.StringUtils
[ERROR] 5 errors found

anybody encounter error like this?
 
Reply | Threaded
Open this post in threaded view
|

Fwd: compile spark 3.1.1 error

Attila Zsolt Piros

Hi!

Zinc cache be cleaned by the shutdown i showed you earlier:
./build/zinc-0.3.15/bin/zinc -shutdown

But as I just seen SPARK-34539: it is absolutely not needed anymore as the standalone zinc is not used by the compiler plugin from v3.0.0.

Regarding make-distribution.sh with hadoop-2.7 I can confirm it is working on my machine:

$ SPARK_HOME=$PWD ./dev/make-distribution.sh --name custom-spark --pip  --tgz -Phive -Phive-thriftserver -Pyarn -Phadoop-2.7  -DskipTests
...
[INFO] Reactor Summary for Spark Project Parent POM 3.1.1:
[INFO]
[INFO] Spark Project Parent POM ........................... SUCCESS [  2.251 s]
[INFO] Spark Project Tags ................................. SUCCESS [  4.513 s]
[INFO] Spark Project Sketch ............................... SUCCESS [  4.878 s]
[INFO] Spark Project Local DB ............................. SUCCESS [  1.259 s]
[INFO] Spark Project Networking ........................... SUCCESS [  3.173 s]
[INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [  1.364 s]
[INFO] Spark Project Unsafe ............................... SUCCESS [  6.672 s]
[INFO] Spark Project Launcher ............................. SUCCESS [  1.782 s]
[INFO] Spark Project Core ................................. SUCCESS [01:48 min]
[INFO] Spark Project ML Local Library ..................... SUCCESS [ 33.861 s]
[INFO] Spark Project GraphX ............................... SUCCESS [ 30.114 s]
[INFO] Spark Project Streaming ............................ SUCCESS [ 42.267 s]
[INFO] Spark Project Catalyst ............................. SUCCESS [02:15 min]
[INFO] Spark Project SQL .................................. SUCCESS [03:05 min]
[INFO] Spark Project ML Library ........................... SUCCESS [02:16 min]
[INFO] Spark Project Tools ................................ SUCCESS [  7.109 s]
[INFO] Spark Project Hive ................................. SUCCESS [01:20 min]
[INFO] Spark Project REPL ................................. SUCCESS [ 20.758 s]
[INFO] Spark Project YARN Shuffle Service ................. SUCCESS [ 10.377 s]
[INFO] Spark Project YARN ................................. SUCCESS [ 47.571 s]
[INFO] Spark Project Hive Thrift Server ................... SUCCESS [ 36.327 s]
[INFO] Spark Project Assembly ............................. SUCCESS [  4.618 s]
[INFO] Kafka 0.10+ Token Provider for Streaming ........... SUCCESS [ 18.895 s]
[INFO] Spark Integration for Kafka 0.10 ................... SUCCESS [ 28.380 s]
[INFO] Kafka 0.10+ Source for Structured Streaming ........ SUCCESS [  02:00 h]
[INFO] Spark Project Examples ............................. SUCCESS [26:28 min]
[INFO] Spark Integration for Kafka 0.10 Assembly .......... SUCCESS [  4.884 s]
[INFO] Spark Avro ......................................... SUCCESS [ 35.016 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  02:43 h
[INFO] Finished at: 2021-03-11T08:12:02+01:00
[INFO] ------------------------------------------------------------------------

...
Creating tar archive
removing 'pyspark-3.1.1' (and everything under it)
+ popd
+ '[' false == true ']'
+ echo 'Skipping building R source package'
Skipping building R source package
+ mkdir /Users/attilazsoltpiros/git/attilapiros/spark/dist/conf
+ cp /Users/attilazsoltpiros/git/attilapiros/spark/conf/fairscheduler.xml.template /Users/attilazsoltpiros/git/attilapiros/spark/conf/log4j.properties.template /Users/attilazsoltpiros/git/attilapiros/spark/conf/metrics.properties.template /Users/attilazsoltpiros/git/attilapiros/spark/conf/spark-defaults.conf.template /Users/attilazsoltpiros/git/attilapiros/spark/conf/spark-env.sh.template /Users/attilazsoltpiros/git/attilapiros/spark/conf/workers.template /Users/attilazsoltpiros/git/attilapiros/spark/dist/conf
+ cp /Users/attilazsoltpiros/git/attilapiros/spark/README.md /Users/attilazsoltpiros/git/attilapiros/spark/dist
+ cp -r /Users/attilazsoltpiros/git/attilapiros/spark/bin /Users/attilazsoltpiros/git/attilapiros/spark/dist
+ cp -r /Users/attilazsoltpiros/git/attilapiros/spark/python /Users/attilazsoltpiros/git/attilapiros/spark/dist
+ '[' true == true ']'
+ rm -f /Users/attilazsoltpiros/git/attilapiros/spark/dist/python/dist/pyspark-3.1.1.tar.gz /Users/attilazsoltpiros/git/attilapiros/spark/dist/python/dist/pyspark-3.2.0.dev0.tar.gz
+ cp -r /Users/attilazsoltpiros/git/attilapiros/spark/sbin /Users/attilazsoltpiros/git/attilapiros/spark/dist
+ '[' -d /Users/attilazsoltpiros/git/attilapiros/spark/R/lib/SparkR ']'
+ mkdir -p /Users/attilazsoltpiros/git/attilapiros/spark/dist/R/lib
+ cp -r /Users/attilazsoltpiros/git/attilapiros/spark/R/lib/SparkR /Users/attilazsoltpiros/git/attilapiros/spark/dist/R/lib
+ cp /Users/attilazsoltpiros/git/attilapiros/spark/R/lib/sparkr.zip /Users/attilazsoltpiros/git/attilapiros/spark/dist/R/lib
+ '[' true == true ']'
+ TARDIR_NAME=spark-3.1.1-bin-custom-spark
+ TARDIR=/Users/attilazsoltpiros/git/attilapiros/spark/spark-3.1.1-bin-custom-spark
+ rm -rf /Users/attilazsoltpiros/git/attilapiros/spark/spark-3.1.1-bin-custom-spark
+ cp -r /Users/attilazsoltpiros/git/attilapiros/spark/dist /Users/attilazsoltpiros/git/attilapiros/spark/spark-3.1.1-bin-custom-spark
+ tar czf spark-3.1.1-bin-custom-spark.tgz -C /Users/attilazsoltpiros/git/attilapiros/spark spark-3.1.1-bin-custom-spark
+ rm -rf /Users/attilazsoltpiros/git/attilapiros/spark/spark-3.1.1-bin-custom-spark

Best Regards,
Attila



On Thu, Mar 11, 2021 at 5:11 AM jiahong li <[hidden email]> wrote:
BTW, how to  clear zinc/nailgun cache, thanks

jiahong li <[hidden email]> 于2021年3月11日周四 下午12:04写道:
Maybe it is my environment cause 

jiahong li <[hidden email]> 于2021年3月11日周四 上午11:14写道:
it not the cause,when i set -Phadoop-2.7 instead of -Dhadoop.version=2.6.0-cdh5.13.1, the same errors come out.

Attila Zsolt Piros <[hidden email]> 于2021年3月10日周三 下午8:56写道:
I see, this must be because of hadoop version you are selecting by using "-Dhadoop.version=2.6.0-cdh5.13.1". 
Spark 3.1.1 only support hadoop-2.7 and hadoop-3.2, at least these two can be given via profiles:  -Phadoop-2.7  and -Phadoop-3.2 (the default). 


On Wed, Mar 10, 2021 at 12:26 PM jiahong li <[hidden email]> wrote:
i use ./build/mvn to compile ,and after execute command :./build/zinc-0.3.15/bin/zinc -shutdown
and execute command like this: /dev/make-distribution.sh --name custom-spark --pip  --tgz -Phive -Phive-thriftserver -Pyarn -Dhadoop.version=2.6.0-cdh5.13.1 -DskipTests
same error appear.
and execute command: ps -ef |grep zinc, there is nothing containe zinc

Attila Zsolt Piros <[hidden email]> 于2021年3月10日周三 下午6:55写道:
hi!

Are you compiling Spark itself?
Do you use "./build/mvn" from the project root? 
If you compiled an other version of Spark before and there the scala version was different then zinc/nailgun could cached the old classes which can cause similar troubles.
In that case this could help:

./build/zinc-0.3.15/bin/zinc -shutdown

Best Regards,
Attila

On Wed, Mar 10, 2021 at 11:27 AM jiahong li <[hidden email]> wrote:
hi, everybody, when i compile spark 3.1.1 from tag v3.1.1 ,encounter error like this:

INFO] --- scala-maven-plugin:4.3.0:compile (scala-compile-first) @ spark-core_2.12 ---
[INFO] Using incremental compilation using Mixed compile order
[INFO] Compiler bridge file: .sbt/1.0/zinc/org.scala-sbt/org.scala-sbt-compiler-bridge_2.12-1.3.1-bin_2.12.10__52.0-1.3.1_20191012T045515.jar
[INFO] compiler plugin: BasicArtifact(com.github.ghik,silencer-plugin_2.12.10,1.6.0,null)
[INFO] Compiling 560 Scala sources and 99 Java sources to git/spark/core/target/scala-2.12/classes ...
[ERROR] [Error] git/spark/core/src/main/scala/org/apache/spark/ui/HttpSecurityFilter.scala:107: type mismatch;
 found   : K where type K
 required: String
[ERROR] [Error] git/spark/core/src/main/scala/org/apache/spark/ui/HttpSecurityFilter.scala:107: value map is not a member of V
[ERROR] [Error] git/spark/core/src/main/scala/org/apache/spark/ui/HttpSecurityFilter.scala:107: missing argument list for method stripXSS in class XssSafeRequest
Unapplied methods are only converted to functions when a function type is expected.
You can make this conversion explicit by writing `stripXSS _` or `stripXSS(_)` instead of `stripXSS`.
[ERROR] [Error] git/spark/core/src/main/scala/org/apache/spark/ui/PagedTable.scala:307: value startsWith is not a member of K
[ERROR] [Error] git/spark/core/src/main/scala/org/apache/spark/util/Utils.scala:580: value toLowerCase is not a member of object org.apache.hadoop.util.StringUtils
[ERROR] 5 errors found

anybody encounter error like this?