Transforming json string in structured streaming problem

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Transforming json string in structured streaming problem

Junfeng Chen
Hi all,

I need to read some string data in json format from kafka, and convert them to dataframe and write to parquet file at last. 
But now I meet some problems. The spark.readStream().json() can only support json file on a specified location, cannot support Dataset<string> like spark.read.json.
I found some potential solution in https://stackoverflow.com/questions/48617474/how-to-convert-json-dataset-to-dataframe-in-spark-structured-streaming , but it needs to construct the StructType, while the structure of my json data is variable. 

So how to solve it? 

Thanks! 


Regard,
Junfeng Chen