How to read multiple libsvm files in Spark?

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

How to read multiple libsvm files in Spark?

Md. Rezaul Karim
I'm experiencing "Exception in thread "main" java.io.IOException: Multiple input paths are not supported for libsvm data" exception while trying to read multiple libsvm files using Spark 2.3.0:

val URLs = spark.read.format("libsvm").load("url_svmlight.tar/url_svmlight/*.svm")

Any other alternatives?
Reply | Threaded
Open this post in threaded view
|

Re: How to read multiple libsvm files in Spark?

Maxim Gekk
Hi,

> Any other alternatives?

Manually form the input path by combining multiple paths via dots. See https://issues.apache.org/jira/browse/SPARK-12086

On Thu, Sep 20, 2018 at 12:47 PM Md. Rezaul Karim <[hidden email]> wrote:
I'm experiencing "Exception in thread "main" java.io.IOException: Multiple input paths are not supported for libsvm data" exception while trying to read multiple libsvm files using Spark 2.3.0:

val URLs = spark.read.format("libsvm").load("url_svmlight.tar/url_svmlight/*.svm")

Any other alternatives?