Spark SELECT statement syntax

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Spark SELECT statement syntax

akashsihag
This post has NOT been accepted by the mailing list yet.
This is in reference with Spark-SQL dataframe "Select query" syntax.

I have the following use case:
 
Earlier, we had a SELECT statement where we were selecting few columns statically from the Dataframe.
Now, due to the new requirement we need to add few dynamic columns in this SELECT statement. These dynamic columns are in a separate list.
 
Solution:
1.) We put the static columns into a separate list.                       // staticColList = List(staticValue1, staticValue2, staticValue3)
2.) Combing both the dynamic and static columns lists into a new list    // castedColList = staticColList ::: dynamicColList
3.) Pass this new list the SELECT statement.          // dataframe.select(castedColList: _*)

This looks little untidy as we first have to create separate lists for static and dynamic columns and then combining them into one for extraction.

Alternative?
Do we have a better way to do it in spark where we can select both static and dynamic columns inside the select statement itself? I mean, something like this - dataframe.select(staticValue1, staticValue2, staticValue3, dynamicList)
Loading...