converting dataframe from one format to another in spark structured streaming

[act_coder]

I am creating a spark structured streaming application and I have a streaming
dataframe which has the below data in it.

{
"name":"sensor1",
"time":"2020-11-27T01:01:00",
"sensorvalue":11.0,
"tag1":"tagvalue"
}
I would like to convert that dataframe into below format.

{
"name":"sensor1",
"value-array":
[
{
"time":"2020-11-27T01:01:00",
"sensorvalue":11.0,
"tag1":"tagvalue"
}
]
}
I tried using mapPartition() / map() method, where I get a row Object and I
tried creating another dataframe in the expected format.

I am able to get the values from row object using row(0) or row(1). But, Is
it possible to put a POJO/schema for that row object (Using Row Encoders())
? That way instead of using row(0), we can use row.getName() ?

val mapDF = incomingDF.map(row =>{
    val name = row(0).toString
    /* val name = row.getName */

  })
I tried using collect_list() function. Since its an aggregate function, I
couldn't use "append" as output mode in my streaming application.



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]