How to read csv in dataframe

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

How to read csv in dataframe

Lehak Dharmani
I am trying to read csv in spark dataframe . My Os = Ubuntu 18.04,
spark-version 2.3.1, python -version 2.7.15
My code :
from pyspark import SparkConf
from pyspark import SparkContext
from pyspark import SQLContext
from pyspark.sql import SparkSession
conf = SparkConf()
sc = SparkContext(conf = conf)
spark =
SparkSession.builder.config(conf=conf).appName('FinancialRecon').getOrCreate()
sqlContext =SQLContext(sc)
df=spark.read.csv('/home/iconnect4/finrecon/test2.csv')
df.show()

Error :
Traceback (most recent call last):
  File "/home/iconnect4/finrecon/scratch_3.py", line 43, in <module>
    df=sqlContext.read.csv('/home/iconnect4/finrecon/test2.csv')
  File "/usr/local/spark/python/lib/pyspark.zip/pyspark/sql/readwriter.py",
line 441, in csv
  File
"/usr/local/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line
1257, in __call__
  File "/usr/local/spark/python/lib/pyspark.zip/pyspark/sql/utils.py", line
79, in deco
pyspark.sql.utils.IllegalArgumentException: 'Illegal pattern component: XXX'

Please help





--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]