Multiple Sources found for csv

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Multiple Sources found for csv

jeffsaremi

I have this line which works in the spark interactive console but it fails in Intellij

Using Spark 2.1.1 in both cases:

Exception in thread "main" java.lang.RuntimeException: Multiple sources found for csv (org.apache.spark.sql.execution.datasources.csv.CSVFileFormat, com.databricks.spark.csv.DefaultSource15), please specify the fully qualified class name.



source:

val ie8df = sqlContext.read.schema(SomeSchema).option("mode","dropmalformed").option("sep", "\t").format("csv").load(somepath)

Reply | Threaded
Open this post in threaded view
|

Re: Multiple Sources found for csv

jeffsaremi

sorry just found this which answers my question:


https://stackoverflow.com/questions/41726340/spark-2-0-csv-error



From: jeff saremi <[hidden email]>
Sent: Tuesday, September 12, 2017 3:38:00 PM
To: [hidden email]
Subject: Multiple Sources found for csv
 

I have this line which works in the spark interactive console but it fails in Intellij

Using Spark 2.1.1 in both cases:

Exception in thread "main" java.lang.RuntimeException: Multiple sources found for csv (org.apache.spark.sql.execution.datasources.csv.CSVFileFormat, com.databricks.spark.csv.DefaultSource15), please specify the fully qualified class name.



source:

val ie8df = sqlContext.read.schema(SomeSchema).option("mode","dropmalformed").option("sep", "\t").format("csv").load(somepath)