Fast Unit Tests

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Fast Unit Tests

marcos rebelo
Hey all,

We are using Scala and SQL heavily, but I have a problem with VERY SLOW Unit Tests. 

Is there a way to do fast Unit Tests on Spark? 

How are you guys going around it?

Best Regards
Marcos Rebelo
Reply | Threaded
Open this post in threaded view
|

Re: Fast Unit Tests

Geoff Von Allmen

I am pretty new to spark/scala myself, but I just recently implemented unit tests to test my transformations/aggregations and such myself.

I’m using the mrpowers spark-fast-tests and spark-daria libraries.

I am also using a JDBC sink in the foreach writer. I’ve mocked the sink to place the generated MYSQL statements in a global object and then I compare the output there to an expected set of mysql statements.

I’m running this with FlatSpec ScalaTests, where my spark inputs are manually generated fixtures for each test case. Everything seems to be running well and it's nice and quick.


On Tue, May 1, 2018 at 8:25 AM, marcos rebelo <[hidden email]> wrote:
Hey all,

We are using Scala and SQL heavily, but I have a problem with VERY SLOW Unit Tests. 

Is there a way to do fast Unit Tests on Spark? 

How are you guys going around it?

Best Regards
Marcos Rebelo

Reply | Threaded
Open this post in threaded view
|

RE: Fast Unit Tests

Yeikel Santana
In reply to this post by marcos rebelo

Can you share a sample test case? How are you doing the unit tests? Are you creating the session in a beforeAll block or similar?

 

As far as I know, if you use spark you will end up with light integration tests rather than “real” unit tests (please correct me if I am wrong).

 

From: marcos rebelo <[hidden email]>
Sent: Tuesday, May 1, 2018 11:25 AM
To: user <[hidden email]>
Subject: Fast Unit Tests

 

Hey all,

 

We are using Scala and SQL heavily, but I have a problem with VERY SLOW Unit Tests. 

 

Is there a way to do fast Unit Tests on Spark? 

 

How are you guys going around it?

 

Best Regards

Marcos Rebelo