pyspark reading lzo in a spitable way

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

pyspark reading lzo in a spitable way

Lavallen Pablo

hi everyone! a question:

someone knows if it's possible to read a lzo compressed file from hdfs to pyspark dataframe directly. 
In a splitable way?

Something like :
    
spark.read.csv("codec...", file.lzo)

all the options I've seen use rdd instead of a DF, and then toDF to get a dataframe


Thanks!!