error , saving dataframe , LEGACY_PASS_PARTITION_BY_AS_OPTIONS

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

error , saving dataframe , LEGACY_PASS_PARTITION_BY_AS_OPTIONS

asma zgolli

Hello , 


I'm using spark 2.4.4 and i keep receiving this error message. Can you please help me identify the problem?


thank you , 

yours sincerely

Asma ZGOLLI

PhD student in data engineering - computer science

PJ:



"main" java.lang.NoSuchMethodError: org.apache.spark.sql.internal.SQLConf$.LEGACY_PASS_PARTITION_BY_AS_OPTIONS()Lorg/apache/spark/internal/config/ConfigEntry;

at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:277)






Reply | Threaded
Open this post in threaded view
|

Re: error , saving dataframe , LEGACY_PASS_PARTITION_BY_AS_OPTIONS

femibyte
Can you post the line of code that’s resulting in that error along with the stack trace ?

Sent from my iPhone

On Nov 13, 2019, at 9:53 AM, asma zgolli <[hidden email]> wrote:



Hello , 


I'm using spark 2.4.4 and i keep receiving this error message. Can you please help me identify the problem?


thank you , 

yours sincerely

Asma ZGOLLI

PhD student in data engineering - computer science

PJ:



"main" java.lang.NoSuchMethodError: org.apache.spark.sql.internal.SQLConf$.LEGACY_PASS_PARTITION_BY_AS_OPTIONS()Lorg/apache/spark/internal/config/ConfigEntry;

at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:277)






Reply | Threaded
Open this post in threaded view
|

Re: error , saving dataframe , LEGACY_PASS_PARTITION_BY_AS_OPTIONS

asma zgolli
the line of code that resulted in that error was : 

ch.cern.sparkmeasure.StageMetrics.saveData 


it also was started while trying to save a dataframe in hbase using hbase spark connector : 

df1.write.options(

      Map(HBaseTableCatalog.tableCatalog -> catalog, HBaseTableCatalog.newTable -> "4"))

      .format("org.apache.hadoop.hbase.spark")

      .save()


the full stack trace is as follows:



Exception in thread "main" java.lang.NoSuchMethodError: org.apache.spark.sql.internal.SQLConf$.LEGACY_PASS_PARTITION_BY_AS_OPTIONS()Lorg/apache/spark/internal/config/ConfigEntry;

at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:277)

at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:271)

at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:229)

at ch.cern.sparkmeasure.StageMetrics.saveData(stagemetrics.scala:297)

...........

...........

at scala.Function0$class.apply$mcV$sp(Function0.scala:34)

at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)

at scala.App$$anonfun$main$1.apply(App.scala:76)

at scala.App$$anonfun$main$1.apply(App.scala:76)

at scala.collection.immutable.List.foreach(List.scala:381)

at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35)

at scala.App$class.main(App.scala:76)


Le mer. 13 nov. 2019 à 16:05, Femi Anthony <[hidden email]> a écrit :
Can you post the line of code that’s resulting in that error along with the stack trace ?

Sent from my iPhone

On Nov 13, 2019, at 9:53 AM, asma zgolli <[hidden email]> wrote:



Hello , 


I'm using spark 2.4.4 and i keep receiving this error message. Can you please help me identify the problem?


thank you , 

yours sincerely

Asma ZGOLLI

PhD student in data engineering - computer science

PJ:



"main" java.lang.NoSuchMethodError: org.apache.spark.sql.internal.SQLConf$.LEGACY_PASS_PARTITION_BY_AS_OPTIONS()Lorg/apache/spark/internal/config/ConfigEntry;

at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:277)








--
Asma ZGOLLI

PhD student in data engineering - computer science

Reply | Threaded
Open this post in threaded view
|

Re: error , saving dataframe , LEGACY_PASS_PARTITION_BY_AS_OPTIONS

Russell Spitzer
In reply to this post by femibyte
My guess would be this is a Spark Version mismatch. The option is added 

https://github.com/apache/spark/commit/df9a50637e2622a15e9af7d837986a0e868878b1

https://issues.apache.org/jira/browse/SPARK-27453

In 2.4.2

I would make sure your Spark installs are all 2.4.4 and that your code is compiled against 2.4.4 spark libs. For example if one node had 2.4.1 or 2.4.0 you would end up with an error.

On Wed, Nov 13, 2019 at 9:05 AM Femi Anthony <[hidden email]> wrote:
Can you post the line of code that’s resulting in that error along with the stack trace ?

Sent from my iPhone

On Nov 13, 2019, at 9:53 AM, asma zgolli <[hidden email]> wrote:



Hello , 


I'm using spark 2.4.4 and i keep receiving this error message. Can you please help me identify the problem?


thank you , 

yours sincerely

Asma ZGOLLI

PhD student in data engineering - computer science

PJ:



"main" java.lang.NoSuchMethodError: org.apache.spark.sql.internal.SQLConf$.LEGACY_PASS_PARTITION_BY_AS_OPTIONS()Lorg/apache/spark/internal/config/ConfigEntry;

at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:277)