Note that Spark never guarantees ordering of columns. There’s nothing in Spark documentation that says that the columns will be ordered a certain way. The proposed solution relies on an implementation detail that might change in future
version of Spark.
Ideally, you shouldn’t rely on Dataframe to maintain order of columns. The question is why do you care about ordering of cols? If order of data is important, then you should put it in an array