Run spark tests on Windows/docker

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Run spark tests on Windows/docker

Shmuel Blitz
Hi,

I'd like to build and run spark tests on my PC.

Build works fine on my Windows machine, but the tests can't run for various reasons.

1. Is it possible to run the tests on a Windows without special magic?
2. If you need some magic, how complicated is it?
3. I thought about running the tests on a docker linux container with the Spark build mounted from the host PC. Has anyone done this? Do you have a recommended docker image to work with?
4. any special consideration I should think of?

Thanks,
Shmuel

--
Shmuel Blitz
Big Data Developer
Email: [hidden email]
www.similarweb.com
Reply | Threaded
Open this post in threaded view
|

Re: Run spark tests on Windows/docker

Shmuel Blitz
Since I got no feedback, I'll try asking differently:

Can anyone point me to any resources regarding how to run the project's tests?
Where can I find a good Docker image that would serve as a YARN cluster for submitting jobs?

Thanks,
Shmuel

On Sun, Sep 16, 2018 at 10:09 PM Shmuel Blitz <[hidden email]> wrote:
Hi,

I'd like to build and run spark tests on my PC.

Build works fine on my Windows machine, but the tests can't run for various reasons.

1. Is it possible to run the tests on a Windows without special magic?
2. If you need some magic, how complicated is it?
3. I thought about running the tests on a docker linux container with the Spark build mounted from the host PC. Has anyone done this? Do you have a recommended docker image to work with?
4. any special consideration I should think of?

Thanks,
Shmuel

--
Shmuel Blitz
Big Data Developer
Email: [hidden email]
www.similarweb.com


--
Shmuel Blitz
Big Data Developer
Email: [hidden email]
www.similarweb.com