withColumn on nested schema

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

withColumn on nested schema

Zsolt Tóth

I'm trying to replace values in a nested column in a JSON-based dataframe using withColumn().

This syntax works for select, filter, etc, giving only the nested "country" column:

but if I do this, it will create a new column with the name "body.payload.country"
df.withColumn('body.payload.country', lit(None))

Also tried with 'body["payload"]["country"]', etc. but no luck. Is it possible to do this somehow?