I have found something similar with my case here https://www.youtube.com/watch?v=EhRHQPCdldI, but unfortunately I don't know if the company Genesys (former AltoCloud) made this API
(StreamPipeline, Heterogenous Pipeline )
I handled this with the above schema code, but I don't know how optimal is.
//apply model val pdf = model.transform(batchDF).selectExpr("CAST(to_json(struct(*)) AS STRING) AS value").write .format("json") .save("anom/" + lm + System.currentTimeMillis()) }).start().awaitTermination() })
1. Therefore, I would like to know if there is any any Spark API for handling such an use case?