OneHotEncoderEstimator - java.lang.NoSuchMethodError: org.apache.spark.sql.Dataset.withColumns

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

OneHotEncoderEstimator - java.lang.NoSuchMethodError: org.apache.spark.sql.Dataset.withColumns

Mina Aslani
Hi,

I get below error when I try to run oneHotEncoderEstimator example. 

Which is this line of the code:
https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/ml/feature/OneHotEncoderEstimator.scala#L348

Exception in thread "streaming-job-executor-0" java.lang.NoSuchMethodError: org.apache.spark.sql.Dataset.withColumns(Lscala/collection/Seq;Lscala/collection/Seq;)Lorg/apache/spark/sql/Dataset;
	at org.apache.spark.ml.feature.OneHotEncoderModel.transform(OneHotEncoderEstimator.scala:348)

Can you please let me know, what is the cause? Any workaround?

Seeing the example in the repo, looks like that at some point it used be running fine. And, now it's not working. Also, oneHotEncoder is deprecated.

I really appreciate your quick response.

Regards,
Mina

Reply | Threaded
Open this post in threaded view
|

Re: OneHotEncoderEstimator - java.lang.NoSuchMethodError: org.apache.spark.sql.Dataset.withColumns

matteuan
Hi,

are you sure Dataset has a method withColumns?

On 15 May 2018 at 16:58, Mina Aslani <[hidden email]> wrote:
Hi,

I get below error when I try to run oneHotEncoderEstimator example. 

Which is this line of the code:
https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/ml/feature/OneHotEncoderEstimator.scala#L348

Exception in thread "streaming-job-executor-0" java.lang.NoSuchMethodError: org.apache.spark.sql.Dataset.withColumns(Lscala/collection/Seq;Lscala/collection/Seq;)Lorg/apache/spark/sql/Dataset;
	at org.apache.spark.ml.feature.OneHotEncoderModel.transform(OneHotEncoderEstimator.scala:348)

Can you please let me know, what is the cause? Any workaround?

Seeing the example in the repo, looks like that at some point it used be running fine. And, now it's not working. Also, oneHotEncoder is deprecated.

I really appreciate your quick response.

Regards,
Mina


Reply | Threaded
Open this post in threaded view
|

Re: OneHotEncoderEstimator - java.lang.NoSuchMethodError: org.apache.spark.sql.Dataset.withColumns

Bryan Cutler
The example works for me, please check your environment and ensure you are using Spark 2.3.0 where OneHotEncoderEstimator was introduced.

On Fri, May 18, 2018 at 12:57 AM, Matteo Cossu <[hidden email]> wrote:
Hi,

are you sure Dataset has a method withColumns?

On 15 May 2018 at 16:58, Mina Aslani <[hidden email]> wrote:
Hi,

I get below error when I try to run oneHotEncoderEstimator example. 

Which is this line of the code:
https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/ml/feature/OneHotEncoderEstimator.scala#L348

Exception in thread "streaming-job-executor-0" java.lang.NoSuchMethodError: org.apache.spark.sql.Dataset.withColumns(Lscala/collection/Seq;Lscala/collection/Seq;)Lorg/apache/spark/sql/Dataset;
	at org.apache.spark.ml.feature.OneHotEncoderModel.transform(OneHotEncoderEstimator.scala:348)

Can you please let me know, what is the cause? Any workaround?

Seeing the example in the repo, looks like that at some point it used be running fine. And, now it's not working. Also, oneHotEncoder is deprecated.

I really appreciate your quick response.

Regards,
Mina