How to read csv in dataframe

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

How to read csv in dataframe

Lehak Dharmani
I am trying to read csv in spark dataframe . My Os = Ubuntu 18.04,
spark-version 2.3.1, python -version 2.7.15
My code :
from pyspark import SparkConf
from pyspark import SparkContext
from pyspark import SQLContext
from pyspark.sql import SparkSession
conf = SparkConf()
sc = SparkContext(conf = conf)
spark =
sqlContext =SQLContext(sc)'/home/iconnect4/finrecon/test2.csv')

Error :
Traceback (most recent call last):
  File "/home/iconnect4/finrecon/", line 43, in <module>'/home/iconnect4/finrecon/test2.csv')
  File "/usr/local/spark/python/lib/",
line 441, in csv
"/usr/local/spark/python/lib/", line
1257, in __call__
  File "/usr/local/spark/python/lib/", line
79, in deco
pyspark.sql.utils.IllegalArgumentException: 'Illegal pattern component: XXX'

Please help

Sent from:

To unsubscribe e-mail: [hidden email]