Is there any good Docker container / compose with spark 2.4+ and YARN 2.8.2+

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Is there any good Docker container / compose with spark 2.4+ and YARN 2.8.2+

Ivan Petrov
Hi,
looking for a ready to use docker-container that has inside:
- spark 2.4 or higher
- yarn 2.8.2 or higher 
I'm looking for a way to submit spark jobs on yarn. 
Reply | Threaded
Open this post in threaded view
|

Re: Is there any good Docker container / compose with spark 2.4+ and YARN 2.8.2+

rmartine
Ivan,

Although this is kubernetes-related docs it might apply to your use case:


There is a script that can create the image for you in spark distribution, it was added in 2.3. So if you downloaded a spark 2.3+ distribution then you will find this script.

On Wed, Sep 16, 2020 at 7:55 AM Ivan Petrov <[hidden email]> wrote:
Hi,
looking for a ready to use docker-container that has inside:
- spark 2.4 or higher
- yarn 2.8.2 or higher 
I'm looking for a way to submit spark jobs on yarn. 


--

Ricardo Martinelli De Oliveira

Data Engineer, AI CoE

Red Hat Brazil

Av. Brigadeiro Faria Lima, 3900

8th floor

[hidden email]    T: <a href="tel:+551135426125" style="color:rgb(0,0,0);margin:0px" target="_blank">+551135426125    
M: <a href="tel:+5511970696531" style="color:rgb(0,0,0);margin:0px" target="_blank">+5511970696531