Iterating all columns in a pyspark dataframe

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Iterating all columns in a pyspark dataframe

Devi P.V
Hi all, 
What is the best approach for iterating all columns in a pyspark dataframe?I want to apply some conditions on all columns in the dataframe. Currently I am using for loop for iteration. Is it a good practice while using Spark and I am using Spark 3.0
Please advice 

Thanks, 
Devi

Reply | Threaded
Open this post in threaded view
|

Re: Iterating all columns in a pyspark dataframe

srowen
Do you need to iterate anything? you can always write a function of
all columns, df.columns. You can operate on a whole Row at a time too.

On Fri, Sep 4, 2020 at 2:11 AM Devi P.V <[hidden email]> wrote:
>
> Hi all,
> What is the best approach for iterating all columns in a pyspark dataframe?I want to apply some conditions on all columns in the dataframe. Currently I am using for loop for iteration. Is it a good practice while using Spark and I am using Spark 3.0
> Please advice
>
> Thanks,
> Devi
>

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]