unit testing for spark code

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

unit testing for spark code

Amit Sharma-2
Hi, can we write unit tests for spark code. Is there any specific framework?


Thanks
Amit
Reply | Threaded
Open this post in threaded view
|

Re: unit testing for spark code

Mich Talebzadeh
coding in Scala or Python?

Are you using any IDE (IntelliJ, PyCharm) 


   view my Linkedin profile

 

Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.

 



On Mon, 22 Mar 2021 at 13:33, Amit Sharma <[hidden email]> wrote:
Hi, can we write unit tests for spark code. Is there any specific framework?


Thanks
Amit
Reply | Threaded
Open this post in threaded view
|

Re: unit testing for spark code

nick.gustafson
In reply to this post by Amit Sharma-2
I've found pytest works well if you're using PySpark. Though if you have a lot of tests, running them all can be pretty slow.

On Mon, Mar 22, 2021 at 6:32 AM Amit Sharma <[hidden email]> wrote:
Hi, can we write unit tests for spark code. Is there any specific framework?


Thanks
Amit
Reply | Threaded
Open this post in threaded view
|

Re: unit testing for spark code

Attila Zsolt Piros
Hi!

Let me draw your attention to Holden's spark-testing-base project.
The documentation is at  https://github.com/holdenk/spark-testing-base/wiki.

As I usually write test for spark internal features I haven't needed to test so high level.
But I am interested about your experiences.

Best regards,
Attila

On Mon, Mar 22, 2021 at 4:34 PM Nicholas Gustafson <[hidden email]> wrote:
I've found pytest works well if you're using PySpark. Though if you have a lot of tests, running them all can be pretty slow.

On Mon, Mar 22, 2021 at 6:32 AM Amit Sharma <[hidden email]> wrote:
Hi, can we write unit tests for spark code. Is there any specific framework?


Thanks
Amit