Spark streaming dataframe extract message to new columns

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Spark streaming dataframe extract message to new columns

zenglong chen
How to extract some message in streaming dataframe and make a new column?
Thanks.
Reply | Threaded
Open this post in threaded view
|

Re: Spark streaming dataframe extract message to new columns

Gourav Sengupta
Hi,

I think that it should be possible to write a query on the streaming data frame and then write the output of the query to S3 or any other sink layer.

Regards,
Gourav Sengupta

On Sat, Aug 10, 2019 at 9:24 AM zenglong chen <[hidden email]> wrote:
How to extract some message in streaming dataframe and make a new column?
Thanks.
Reply | Threaded
Open this post in threaded view
|

Re: Spark streaming dataframe extract message to new columns

Tianlang

Hi,

Do you mean haven a colum A then you want to extract A1 and A2 from A ?

like

Cloumn A value   123456,2019-08-07

A1 value is 123456

A2 value is 2019-08-07

If that's the case you can use df.select like this

df.select(split('A)(0) as "A1", split('A)(1) as "A2")


Good Luck

在 2019/8/12 下午8:51, Gourav Sengupta 写道:
Hi,

I think that it should be possible to write a query on the streaming data frame and then write the output of the query to S3 or any other sink layer.

Regards,
Gourav Sengupta

On Sat, Aug 10, 2019 at 9:24 AM zenglong chen <[hidden email]> wrote:
How to extract some message in streaming dataframe and make a new column?
Thanks.