CSV parser - how to parse column containing json data

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

CSV parser - how to parse column containing json data

Nirav Patel
Is there a way to parse csv file with some column in middle containing json data structure?

"a",102,"c","{"x":"xx","y":false,"z":123}","d","e",102.2


Thanks,
Nirav



What's New with Xactly

        
Reply | Threaded
Open this post in threaded view
|

Re: CSV parser - how to parse column containing json data

Brandon Geise

If you know your json schema you can create a struct and then apply that using from_json:

 

val json_schema = StructType(Array(StructField(“x”, StringType, true), StructField(“y”, StringType, true), StructField(“z”, IntegerType, true)))

 

.withColumn("_c3", from_json(col("_c3_signals"),json_schema))

 

From: Nirav Patel <[hidden email]>
Date: Thursday, August 30, 2018 at 7:19 PM
To: spark users <[hidden email]>
Subject: CSV parser - how to parse column containing json data

 

Is there a way to parse csv file with some column in middle containing json data structure?

 

"a",102,"c","{"x":"xx","y":false,"z":123}","d","e",102.2

 

 

Thanks,

Nirav




What's New with Xactly

https://www.xactlycorp.com/wp-content/uploads/2017/09/insta.png  https://www.xactlycorp.com/wp-content/uploads/2017/09/linkedin.png  https://www.xactlycorp.com/wp-content/uploads/2017/09/twitter.png  https://www.xactlycorp.com/wp-content/uploads/2017/09/facebook.png  https://www.xactlycorp.com/wp-content/uploads/2017/09/youtube.png