Integration testing Framework Spark SQL Scala

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Integration testing Framework Spark SQL Scala

Ruijing Li
Hi all,

I’m interested in hearing the community’s thoughts on best practices to do integration testing for spark sql jobs. We run a lot of our jobs with cloud infrastructure and hdfs - this makes debugging a challenge for us, especially with problems that don’t occur from just initializing a sparksession locally or testing with spark-shell. Ideally, we’d like some sort of docker container emulating hdfs and spark cluster mode, that you can run locally. 

Any test framework, tips, or examples people can share? Thanks!
--
Cheers,
Ruijing Li
Reply | Threaded
Open this post in threaded view
|

Re: Integration testing Framework Spark SQL Scala

Ruijing Li
Just wanted to follow up on this. If anyone has any advice, I’d be interested in learning more!

On Thu, Feb 20, 2020 at 6:09 PM Ruijing Li <[hidden email]> wrote:
Hi all,

I’m interested in hearing the community’s thoughts on best practices to do integration testing for spark sql jobs. We run a lot of our jobs with cloud infrastructure and hdfs - this makes debugging a challenge for us, especially with problems that don’t occur from just initializing a sparksession locally or testing with spark-shell. Ideally, we’d like some sort of docker container emulating hdfs and spark cluster mode, that you can run locally. 

Any test framework, tips, or examples people can share? Thanks!
--
Cheers,
Ruijing Li
--
Cheers,
Ruijing Li