Convert scientific notation DecimalType

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

Convert scientific notation DecimalType

dimitris plakas
Hello everyone,

I am new in Pyspark and i am facing a problem in casting some values in DecimalType. To clarify my question i present an example.

i have a dataframe in which i store my data which are some trajectories the dataframe looks like

Id | Trajectory

id1 | [ [x1, y1, t1], [x2, y2, t2], .......[xn, yn, tn] ]
id2 | [ [x1, y1, t1], [x2, y2, t2], .......[xn, yn, tn] ]

So for each ID i have a trajectory after applying group by in a previous dataframe.
x= lon
t=time (in seconds).

When i store the data in the dataframe the time is displayed in scientific notation. Is there any way to convert the t in DecimalType(10,0) ?

I tried to convert it in the beggining but i get an error when i try to store the trajectories because the column Trajectory is ArrayType(ArrayType(FloatType() ) ).