Running spark code using wheel file

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Running spark code using wheel file

Sachit Murarka
Hi All,

I have created a wheel file and I am using the following command to run the spark job:

spark-submit --py-files application.whl main_flow.py

My application is unable to reference the modules. Do I need to do the pip install of the wheel first?

Kind Regards,
Sachit Murarka