|
|
|
|
hi, everybody, when i compile spark 3.1.1 from tag v3.1.1 ,encounter error like this:
INFO] --- scala-maven-plugin:4.3.0:compile (scala-compile-first) @ spark-core_2.12 ---
[INFO] Using incremental compilation using Mixed compile order
[INFO] Compiler bridge file: .sbt/1.0/zinc/org.scala-sbt/org.scala-sbt-compiler-bridge_2.12-1.3.1-bin_2.12.10__52.0-1.3.1_20191012T045515.jar
[INFO] compiler plugin: BasicArtifact(com.github.ghik,silencer-plugin_2.12.10,1.6.0,null)
[INFO] Compiling 560 Scala sources and 99 Java sources to git/spark/core/target/scala-2.12/classes ...
[ERROR] [Error] git/spark/core/src/main/scala/org/apache/spark/ui/HttpSecurityFilter.scala:107: type mismatch;
found : K where type K
required: String
[ERROR] [Error] git/spark/core/src/main/scala/org/apache/spark/ui/HttpSecurityFilter.scala:107: value map is not a member of V
[ERROR] [Error] git/spark/core/src/main/scala/org/apache/spark/ui/HttpSecurityFilter.scala:107: missing argument list for method stripXSS in class XssSafeRequest
Unapplied methods are only converted to functions when a function type is expected.
You can make this conversion explicit by writing `stripXSS _` or `stripXSS(_)` instead of `stripXSS`.
[ERROR] [Error] git/spark/core/src/main/scala/org/apache/spark/ui/PagedTable.scala:307: value startsWith is not a member of K
[ERROR] [Error] git/spark/core/src/main/scala/org/apache/spark/util/Utils.scala:580: value toLowerCase is not a member of object org.apache.hadoop.util.StringUtils
[ERROR] 5 errors found
anybody encounter error like this?
|
|
|
hi!
Are you compiling Spark itself?
Do you use "./build/mvn" from the project root?
If you compiled an other version of Spark before and there the scala version was different then zinc/nailgun could cached the old classes which can cause similar troubles.
In that case this could help:
./build/zinc-0.3.15/bin/zinc -shutdown
Best Regards,
Attila
hi, everybody, when i compile spark 3.1.1 from tag v3.1.1 ,encounter error like this:
INFO] --- scala-maven-plugin:4.3.0:compile (scala-compile-first) @ spark-core_2.12 ---
[INFO] Using incremental compilation using Mixed compile order
[INFO] Compiler bridge file: .sbt/1.0/zinc/org.scala-sbt/org.scala-sbt-compiler-bridge_2.12-1.3.1-bin_2.12.10__52.0-1.3.1_20191012T045515.jar
[INFO] compiler plugin: BasicArtifact(com.github.ghik,silencer-plugin_2.12.10,1.6.0,null)
[INFO] Compiling 560 Scala sources and 99 Java sources to git/spark/core/target/scala-2.12/classes ...
[ERROR] [Error] git/spark/core/src/main/scala/org/apache/spark/ui/HttpSecurityFilter.scala:107: type mismatch;
found : K where type K
required: String
[ERROR] [Error] git/spark/core/src/main/scala/org/apache/spark/ui/HttpSecurityFilter.scala:107: value map is not a member of V
[ERROR] [Error] git/spark/core/src/main/scala/org/apache/spark/ui/HttpSecurityFilter.scala:107: missing argument list for method stripXSS in class XssSafeRequest
Unapplied methods are only converted to functions when a function type is expected.
You can make this conversion explicit by writing `stripXSS _` or `stripXSS(_)` instead of `stripXSS`.
[ERROR] [Error] git/spark/core/src/main/scala/org/apache/spark/ui/PagedTable.scala:307: value startsWith is not a member of K
[ERROR] [Error] git/spark/core/src/main/scala/org/apache/spark/util/Utils.scala:580: value toLowerCase is not a member of object org.apache.hadoop.util.StringUtils
[ERROR] 5 errors found
anybody encounter error like this?
|
|
|
i use ./build/mvn to compile ,and after execute command : ./build/zinc-0.3.15/bin/zinc -shutdownand execute command like this: /dev/make-distribution.sh --name custom-spark --pip --tgz -Phive -Phive-thriftserver -Pyarn -Dhadoop.version=2.6.0-cdh5.13.1 -DskipTests same error appear. and execute command: ps -ef |grep zinc, there is nothing containe zinc hi!
Are you compiling Spark itself?
Do you use "./build/mvn" from the project root?
If you compiled an other version of Spark before and there the scala version was different then zinc/nailgun could cached the old classes which can cause similar troubles.
In that case this could help:
./build/zinc-0.3.15/bin/zinc -shutdown
Best Regards,
Attila
hi, everybody, when i compile spark 3.1.1 from tag v3.1.1 ,encounter error like this:
INFO] --- scala-maven-plugin:4.3.0:compile (scala-compile-first) @ spark-core_2.12 ---
[INFO] Using incremental compilation using Mixed compile order
[INFO] Compiler bridge file: .sbt/1.0/zinc/org.scala-sbt/org.scala-sbt-compiler-bridge_2.12-1.3.1-bin_2.12.10__52.0-1.3.1_20191012T045515.jar
[INFO] compiler plugin: BasicArtifact(com.github.ghik,silencer-plugin_2.12.10,1.6.0,null)
[INFO] Compiling 560 Scala sources and 99 Java sources to git/spark/core/target/scala-2.12/classes ...
[ERROR] [Error] git/spark/core/src/main/scala/org/apache/spark/ui/HttpSecurityFilter.scala:107: type mismatch;
found : K where type K
required: String
[ERROR] [Error] git/spark/core/src/main/scala/org/apache/spark/ui/HttpSecurityFilter.scala:107: value map is not a member of V
[ERROR] [Error] git/spark/core/src/main/scala/org/apache/spark/ui/HttpSecurityFilter.scala:107: missing argument list for method stripXSS in class XssSafeRequest
Unapplied methods are only converted to functions when a function type is expected.
You can make this conversion explicit by writing `stripXSS _` or `stripXSS(_)` instead of `stripXSS`.
[ERROR] [Error] git/spark/core/src/main/scala/org/apache/spark/ui/PagedTable.scala:307: value startsWith is not a member of K
[ERROR] [Error] git/spark/core/src/main/scala/org/apache/spark/util/Utils.scala:580: value toLowerCase is not a member of object org.apache.hadoop.util.StringUtils
[ERROR] 5 errors found
anybody encounter error like this?
|
|
|
I see, this must be because of hadoop version you are selecting by using "-Dhadoop.version=2.6.0-cdh5.13.1". Spark 3.1.1 only support hadoop-2.7 and hadoop-3.2, at least these two can be given via profiles: -Phadoop-2.7 and -Phadoop-3.2 (the default).
i use ./build/mvn to compile ,and after execute command : ./build/zinc-0.3.15/bin/zinc -shutdownand execute command like this: /dev/make-distribution.sh --name custom-spark --pip --tgz -Phive -Phive-thriftserver -Pyarn -Dhadoop.version=2.6.0-cdh5.13.1 -DskipTests same error appear. and execute command: ps -ef |grep zinc, there is nothing containe zinc
hi!
Are you compiling Spark itself?
Do you use "./build/mvn" from the project root?
If you compiled an other version of Spark before and there the scala version was different then zinc/nailgun could cached the old classes which can cause similar troubles.
In that case this could help:
./build/zinc-0.3.15/bin/zinc -shutdown
Best Regards,
Attila
hi, everybody, when i compile spark 3.1.1 from tag v3.1.1 ,encounter error like this:
INFO] --- scala-maven-plugin:4.3.0:compile (scala-compile-first) @ spark-core_2.12 ---
[INFO] Using incremental compilation using Mixed compile order
[INFO] Compiler bridge file: .sbt/1.0/zinc/org.scala-sbt/org.scala-sbt-compiler-bridge_2.12-1.3.1-bin_2.12.10__52.0-1.3.1_20191012T045515.jar
[INFO] compiler plugin: BasicArtifact(com.github.ghik,silencer-plugin_2.12.10,1.6.0,null)
[INFO] Compiling 560 Scala sources and 99 Java sources to git/spark/core/target/scala-2.12/classes ...
[ERROR] [Error] git/spark/core/src/main/scala/org/apache/spark/ui/HttpSecurityFilter.scala:107: type mismatch;
found : K where type K
required: String
[ERROR] [Error] git/spark/core/src/main/scala/org/apache/spark/ui/HttpSecurityFilter.scala:107: value map is not a member of V
[ERROR] [Error] git/spark/core/src/main/scala/org/apache/spark/ui/HttpSecurityFilter.scala:107: missing argument list for method stripXSS in class XssSafeRequest
Unapplied methods are only converted to functions when a function type is expected.
You can make this conversion explicit by writing `stripXSS _` or `stripXSS(_)` instead of `stripXSS`.
[ERROR] [Error] git/spark/core/src/main/scala/org/apache/spark/ui/PagedTable.scala:307: value startsWith is not a member of K
[ERROR] [Error] git/spark/core/src/main/scala/org/apache/spark/util/Utils.scala:580: value toLowerCase is not a member of object org.apache.hadoop.util.StringUtils
[ERROR] 5 errors found
anybody encounter error like this?
|
|
|
it not the cause,when i set -Phadoop-2.7 instead of -Dhadoop.version=2.6.0-cdh5.13.1, the same errors come out. I see, this must be because of hadoop version you are selecting by using "-Dhadoop.version=2.6.0-cdh5.13.1". Spark 3.1.1 only support hadoop-2.7 and hadoop-3.2, at least these two can be given via profiles: -Phadoop-2.7 and -Phadoop-3.2 (the default).
i use ./build/mvn to compile ,and after execute command : ./build/zinc-0.3.15/bin/zinc -shutdownand execute command like this: /dev/make-distribution.sh --name custom-spark --pip --tgz -Phive -Phive-thriftserver -Pyarn -Dhadoop.version=2.6.0-cdh5.13.1 -DskipTests same error appear. and execute command: ps -ef |grep zinc, there is nothing containe zinc
hi!
Are you compiling Spark itself?
Do you use "./build/mvn" from the project root?
If you compiled an other version of Spark before and there the scala version was different then zinc/nailgun could cached the old classes which can cause similar troubles.
In that case this could help:
./build/zinc-0.3.15/bin/zinc -shutdown
Best Regards,
Attila
hi, everybody, when i compile spark 3.1.1 from tag v3.1.1 ,encounter error like this:
INFO] --- scala-maven-plugin:4.3.0:compile (scala-compile-first) @ spark-core_2.12 ---
[INFO] Using incremental compilation using Mixed compile order
[INFO] Compiler bridge file: .sbt/1.0/zinc/org.scala-sbt/org.scala-sbt-compiler-bridge_2.12-1.3.1-bin_2.12.10__52.0-1.3.1_20191012T045515.jar
[INFO] compiler plugin: BasicArtifact(com.github.ghik,silencer-plugin_2.12.10,1.6.0,null)
[INFO] Compiling 560 Scala sources and 99 Java sources to git/spark/core/target/scala-2.12/classes ...
[ERROR] [Error] git/spark/core/src/main/scala/org/apache/spark/ui/HttpSecurityFilter.scala:107: type mismatch;
found : K where type K
required: String
[ERROR] [Error] git/spark/core/src/main/scala/org/apache/spark/ui/HttpSecurityFilter.scala:107: value map is not a member of V
[ERROR] [Error] git/spark/core/src/main/scala/org/apache/spark/ui/HttpSecurityFilter.scala:107: missing argument list for method stripXSS in class XssSafeRequest
Unapplied methods are only converted to functions when a function type is expected.
You can make this conversion explicit by writing `stripXSS _` or `stripXSS(_)` instead of `stripXSS`.
[ERROR] [Error] git/spark/core/src/main/scala/org/apache/spark/ui/PagedTable.scala:307: value startsWith is not a member of K
[ERROR] [Error] git/spark/core/src/main/scala/org/apache/spark/util/Utils.scala:580: value toLowerCase is not a member of object org.apache.hadoop.util.StringUtils
[ERROR] 5 errors found
anybody encounter error like this?
|
|
|
Maybe it is my environment cause it not the cause,when i set -Phadoop-2.7 instead of -Dhadoop.version=2.6.0-cdh5.13.1, the same errors come out.
I see, this must be because of hadoop version you are selecting by using "-Dhadoop.version=2.6.0-cdh5.13.1". Spark 3.1.1 only support hadoop-2.7 and hadoop-3.2, at least these two can be given via profiles: -Phadoop-2.7 and -Phadoop-3.2 (the default).
i use ./build/mvn to compile ,and after execute command : ./build/zinc-0.3.15/bin/zinc -shutdownand execute command like this: /dev/make-distribution.sh --name custom-spark --pip --tgz -Phive -Phive-thriftserver -Pyarn -Dhadoop.version=2.6.0-cdh5.13.1 -DskipTests same error appear. and execute command: ps -ef |grep zinc, there is nothing containe zinc
hi!
Are you compiling Spark itself?
Do you use "./build/mvn" from the project root?
If you compiled an other version of Spark before and there the scala version was different then zinc/nailgun could cached the old classes which can cause similar troubles.
In that case this could help:
./build/zinc-0.3.15/bin/zinc -shutdown
Best Regards,
Attila
hi, everybody, when i compile spark 3.1.1 from tag v3.1.1 ,encounter error like this:
INFO] --- scala-maven-plugin:4.3.0:compile (scala-compile-first) @ spark-core_2.12 ---
[INFO] Using incremental compilation using Mixed compile order
[INFO] Compiler bridge file: .sbt/1.0/zinc/org.scala-sbt/org.scala-sbt-compiler-bridge_2.12-1.3.1-bin_2.12.10__52.0-1.3.1_20191012T045515.jar
[INFO] compiler plugin: BasicArtifact(com.github.ghik,silencer-plugin_2.12.10,1.6.0,null)
[INFO] Compiling 560 Scala sources and 99 Java sources to git/spark/core/target/scala-2.12/classes ...
[ERROR] [Error] git/spark/core/src/main/scala/org/apache/spark/ui/HttpSecurityFilter.scala:107: type mismatch;
found : K where type K
required: String
[ERROR] [Error] git/spark/core/src/main/scala/org/apache/spark/ui/HttpSecurityFilter.scala:107: value map is not a member of V
[ERROR] [Error] git/spark/core/src/main/scala/org/apache/spark/ui/HttpSecurityFilter.scala:107: missing argument list for method stripXSS in class XssSafeRequest
Unapplied methods are only converted to functions when a function type is expected.
You can make this conversion explicit by writing `stripXSS _` or `stripXSS(_)` instead of `stripXSS`.
[ERROR] [Error] git/spark/core/src/main/scala/org/apache/spark/ui/PagedTable.scala:307: value startsWith is not a member of K
[ERROR] [Error] git/spark/core/src/main/scala/org/apache/spark/util/Utils.scala:580: value toLowerCase is not a member of object org.apache.hadoop.util.StringUtils
[ERROR] 5 errors found
anybody encounter error like this?
|
|
|
Hi! Zinc cache be cleaned by the shutdown i showed you earlier: ./build/zinc-0.3.15/bin/zinc -shutdown
But as I just seen SPARK-34539: it is absolutely not needed anymore as the standalone zinc is not used by the compiler plugin from v3.0.0.
Regarding make-distribution.sh with hadoop-2.7 I can confirm it is working on my machine:
$ SPARK_HOME=$PWD ./dev/make-distribution.sh --name custom-spark --pip --tgz -Phive -Phive-thriftserver -Pyarn -Phadoop-2.7 -DskipTests
...
[INFO] Reactor Summary for Spark Project Parent POM 3.1.1:
[INFO]
[INFO] Spark Project Parent POM ........................... SUCCESS [ 2.251 s]
[INFO] Spark Project Tags ................................. SUCCESS [ 4.513 s]
[INFO] Spark Project Sketch ............................... SUCCESS [ 4.878 s]
[INFO] Spark Project Local DB ............................. SUCCESS [ 1.259 s]
[INFO] Spark Project Networking ........................... SUCCESS [ 3.173 s]
[INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [ 1.364 s]
[INFO] Spark Project Unsafe ............................... SUCCESS [ 6.672 s]
[INFO] Spark Project Launcher ............................. SUCCESS [ 1.782 s]
[INFO] Spark Project Core ................................. SUCCESS [01:48 min]
[INFO] Spark Project ML Local Library ..................... SUCCESS [ 33.861 s]
[INFO] Spark Project GraphX ............................... SUCCESS [ 30.114 s]
[INFO] Spark Project Streaming ............................ SUCCESS [ 42.267 s]
[INFO] Spark Project Catalyst ............................. SUCCESS [02:15 min]
[INFO] Spark Project SQL .................................. SUCCESS [03:05 min]
[INFO] Spark Project ML Library ........................... SUCCESS [02:16 min]
[INFO] Spark Project Tools ................................ SUCCESS [ 7.109 s]
[INFO] Spark Project Hive ................................. SUCCESS [01:20 min]
[INFO] Spark Project REPL ................................. SUCCESS [ 20.758 s]
[INFO] Spark Project YARN Shuffle Service ................. SUCCESS [ 10.377 s]
[INFO] Spark Project YARN ................................. SUCCESS [ 47.571 s]
[INFO] Spark Project Hive Thrift Server ................... SUCCESS [ 36.327 s]
[INFO] Spark Project Assembly ............................. SUCCESS [ 4.618 s]
[INFO] Kafka 0.10+ Token Provider for Streaming ........... SUCCESS [ 18.895 s]
[INFO] Spark Integration for Kafka 0.10 ................... SUCCESS [ 28.380 s]
[INFO] Kafka 0.10+ Source for Structured Streaming ........ SUCCESS [ 02:00 h]
[INFO] Spark Project Examples ............................. SUCCESS [26:28 min]
[INFO] Spark Integration for Kafka 0.10 Assembly .......... SUCCESS [ 4.884 s]
[INFO] Spark Avro ......................................... SUCCESS [ 35.016 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 02:43 h
[INFO] Finished at: 2021-03-11T08:12:02+01:00
[INFO] ------------------------------------------------------------------------
...
Creating tar archive
removing 'pyspark-3.1.1' (and everything under it)
+ popd
+ '[' false == true ']'
+ echo 'Skipping building R source package'
Skipping building R source package
+ mkdir /Users/attilazsoltpiros/git/attilapiros/spark/dist/conf
+ cp /Users/attilazsoltpiros/git/attilapiros/spark/conf/fairscheduler.xml.template /Users/attilazsoltpiros/git/attilapiros/spark/conf/log4j.properties.template /Users/attilazsoltpiros/git/attilapiros/spark/conf/metrics.properties.template /Users/attilazsoltpiros/git/attilapiros/spark/conf/spark-defaults.conf.template /Users/attilazsoltpiros/git/attilapiros/spark/conf/spark-env.sh.template /Users/attilazsoltpiros/git/attilapiros/spark/conf/workers.template /Users/attilazsoltpiros/git/attilapiros/spark/dist/conf
+ cp /Users/attilazsoltpiros/git/attilapiros/spark/README.md /Users/attilazsoltpiros/git/attilapiros/spark/dist
+ cp -r /Users/attilazsoltpiros/git/attilapiros/spark/bin /Users/attilazsoltpiros/git/attilapiros/spark/dist
+ cp -r /Users/attilazsoltpiros/git/attilapiros/spark/python /Users/attilazsoltpiros/git/attilapiros/spark/dist
+ '[' true == true ']'
+ rm -f /Users/attilazsoltpiros/git/attilapiros/spark/dist/python/dist/pyspark-3.1.1.tar.gz /Users/attilazsoltpiros/git/attilapiros/spark/dist/python/dist/pyspark-3.2.0.dev0.tar.gz
+ cp -r /Users/attilazsoltpiros/git/attilapiros/spark/sbin /Users/attilazsoltpiros/git/attilapiros/spark/dist
+ '[' -d /Users/attilazsoltpiros/git/attilapiros/spark/R/lib/SparkR ']'
+ mkdir -p /Users/attilazsoltpiros/git/attilapiros/spark/dist/R/lib
+ cp -r /Users/attilazsoltpiros/git/attilapiros/spark/R/lib/SparkR /Users/attilazsoltpiros/git/attilapiros/spark/dist/R/lib
+ cp /Users/attilazsoltpiros/git/attilapiros/spark/R/lib/sparkr.zip /Users/attilazsoltpiros/git/attilapiros/spark/dist/R/lib
+ '[' true == true ']'
+ TARDIR_NAME=spark-3.1.1-bin-custom-spark
+ TARDIR=/Users/attilazsoltpiros/git/attilapiros/spark/spark-3.1.1-bin-custom-spark
+ rm -rf /Users/attilazsoltpiros/git/attilapiros/spark/spark-3.1.1-bin-custom-spark
+ cp -r /Users/attilazsoltpiros/git/attilapiros/spark/dist /Users/attilazsoltpiros/git/attilapiros/spark/spark-3.1.1-bin-custom-spark
+ tar czf spark-3.1.1-bin-custom-spark.tgz -C /Users/attilazsoltpiros/git/attilapiros/spark spark-3.1.1-bin-custom-spark
+ rm -rf /Users/attilazsoltpiros/git/attilapiros/spark/spark-3.1.1-bin-custom-spark
Best Regards, Attila
BTW, how to clear zinc/nailgun cache, thanks
Maybe it is my environment cause
it not the cause,when i set -Phadoop-2.7 instead of -Dhadoop.version=2.6.0-cdh5.13.1, the same errors come out.
I see, this must be because of hadoop version you are selecting by using "-Dhadoop.version=2.6.0-cdh5.13.1". Spark 3.1.1 only support hadoop-2.7 and hadoop-3.2, at least these two can be given via profiles: -Phadoop-2.7 and -Phadoop-3.2 (the default).
i use ./build/mvn to compile ,and after execute command : ./build/zinc-0.3.15/bin/zinc -shutdownand execute command like this: /dev/make-distribution.sh --name custom-spark --pip --tgz -Phive -Phive-thriftserver -Pyarn -Dhadoop.version=2.6.0-cdh5.13.1 -DskipTests same error appear. and execute command: ps -ef |grep zinc, there is nothing containe zinc
hi!
Are you compiling Spark itself?
Do you use "./build/mvn" from the project root?
If you compiled an other version of Spark before and there the scala version was different then zinc/nailgun could cached the old classes which can cause similar troubles.
In that case this could help:
./build/zinc-0.3.15/bin/zinc -shutdown
Best Regards,
Attila
hi, everybody, when i compile spark 3.1.1 from tag v3.1.1 ,encounter error like this:
INFO] --- scala-maven-plugin:4.3.0:compile (scala-compile-first) @ spark-core_2.12 ---
[INFO] Using incremental compilation using Mixed compile order
[INFO] Compiler bridge file: .sbt/1.0/zinc/org.scala-sbt/org.scala-sbt-compiler-bridge_2.12-1.3.1-bin_2.12.10__52.0-1.3.1_20191012T045515.jar
[INFO] compiler plugin: BasicArtifact(com.github.ghik,silencer-plugin_2.12.10,1.6.0,null)
[INFO] Compiling 560 Scala sources and 99 Java sources to git/spark/core/target/scala-2.12/classes ...
[ERROR] [Error] git/spark/core/src/main/scala/org/apache/spark/ui/HttpSecurityFilter.scala:107: type mismatch;
found : K where type K
required: String
[ERROR] [Error] git/spark/core/src/main/scala/org/apache/spark/ui/HttpSecurityFilter.scala:107: value map is not a member of V
[ERROR] [Error] git/spark/core/src/main/scala/org/apache/spark/ui/HttpSecurityFilter.scala:107: missing argument list for method stripXSS in class XssSafeRequest
Unapplied methods are only converted to functions when a function type is expected.
You can make this conversion explicit by writing `stripXSS _` or `stripXSS(_)` instead of `stripXSS`.
[ERROR] [Error] git/spark/core/src/main/scala/org/apache/spark/ui/PagedTable.scala:307: value startsWith is not a member of K
[ERROR] [Error] git/spark/core/src/main/scala/org/apache/spark/util/Utils.scala:580: value toLowerCase is not a member of object org.apache.hadoop.util.StringUtils
[ERROR] 5 errors found
anybody encounter error like this?
|
|
Locked
