Struct Type

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Struct Type

khajaasmath786
Hi,

I have following schema in dataframe and I want to extract key which matches as MaxSpeed from the array and it's corresponding value of the key.

|-- tags: array (nullable = true)
 |    |-- element: struct (containsNull = true)
 |    |    |-- key: string (nullable = true)
 |    |    |-- value: string (nullable = true)

is there any way to achieve it in dataframe? 

Thanks,
Asmath
Reply | Threaded
Open this post in threaded view
|

Re: Struct Type

Jacek Laskowski
Hi,

Use explode function, filter operator and collect_list function.

Or "heavier" flatMap.

Pozdrawiam,
Jacek Laskowski
----
Spark Structured Streaming https://bit.ly/spark-structured-streaming
Mastering Apache Spark 2 https://bit.ly/mastering-apache-spark

On Fri, Nov 17, 2017 at 6:06 PM, KhajaAsmath Mohammed <[hidden email]> wrote:
Hi,

I have following schema in dataframe and I want to extract key which matches as MaxSpeed from the array and it's corresponding value of the key.

|-- tags: array (nullable = true)
 |    |-- element: struct (containsNull = true)
 |    |    |-- key: string (nullable = true)
 |    |    |-- value: string (nullable = true)

is there any way to achieve it in dataframe? 

Thanks,
Asmath