Insert a pyspark dataframe in postgresql

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Insert a pyspark dataframe in postgresql

dimitris plakas
Hello everyone here is a case that i am facing,

i have a pyspark application that as it's last step is to create a pyspark dataframe with two columns
(column1, column2). This dataframe has only one row and i want this row to be inserted in a postgres db table. In every run this line in the dataframe may be different or the same. 

I want after 10 runs to have 10 rows in my postgres table, so i want to insert that dataframe in a postgres table in every run. What i have done up to now is to use the below code but it doesn't insert the new row after every run of my pyspark application, it just overwrites the old row.

Here is my code:
test_write_df.write.mode('append').options(url='jdbc:postgresql://localhost:5432/Test_Db', dbtable='test_write_df',driver='org.postgresql.Driver',user='postgres', password='my_password')
Can you please consult me if is this feasible to be happened and/or how to achieve this?

Thank you in advance