SpecificColumnarIterator has grown past JVM limit of 0xFFF

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

SpecificColumnarIterator has grown past JVM limit of 0xFFF

Md. Rezaul Karim
Dear All,

I was training the RandomForest with an input dataset having 20,000 columns and 12,000 rows.
But when I start the training, it shows an exception:

Constant pool for class org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificColumnarIterator has grown past JVM limit of 0xFFF

I understand that the current implementation cannot handle so many columns. However, I was still wondering if there's any workaround to handle a dataset like this?
 




Kind regards,
_________________________________

Md. Rezaul Karim, BSc, MSc

Research Scientist, Fraunhofer FIT, Germany
PhD Researcher, Information Systems, RWTH Aachen University, Germany

Email: [hidden email]
Phone: <a href="tel:%2B%2B49%20241%2080%2021527" target="_blank">+49 241 80 21527

Web: http://www.reza-analytics.eu/index.html