Where do the executors get my app jar from?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Where do the executors get my app jar from?

James Yu
Hi,

When I spark submit a Spark app with my app jar located in S3, obviously the Driver will download the jar from the s3 location.  What is not clear to me is: where do the Executors get the jar from?  From the same s3 location, or somehow from the Driver, or they don't need the jar?

Thanks in advance for explanation.

James
Reply | Threaded
Open this post in threaded view
|

Re: Where do the executors get my app jar from?

Russell Spitzer
The driver hosts a file server which the executors download the jar from.

On Thu, Aug 13, 2020, 5:33 PM James Yu <[hidden email]> wrote:
Hi,

When I spark submit a Spark app with my app jar located in S3, obviously the Driver will download the jar from the s3 location.  What is not clear to me is: where do the Executors get the jar from?  From the same s3 location, or somehow from the Driver, or they don't need the jar?

Thanks in advance for explanation.

James
Reply | Threaded
Open this post in threaded view
|

Re: Where do the executors get my app jar from?

Russell Spitzer
Looking back at the code

All --jar Args and such run through


Which calls 


Which places local jars on the driver hosted file server and just leaves Remote Jars as is with the path for executors to access them

On Thu, Aug 13, 2020 at 11:01 PM Russell Spitzer <[hidden email]> wrote:
The driver hosts a file server which the executors download the jar from.

On Thu, Aug 13, 2020, 5:33 PM James Yu <[hidden email]> wrote:
Hi,

When I spark submit a Spark app with my app jar located in S3, obviously the Driver will download the jar from the s3 location.  What is not clear to me is: where do the Executors get the jar from?  From the same s3 location, or somehow from the Driver, or they don't need the jar?

Thanks in advance for explanation.

James
Reply | Threaded
Open this post in threaded view
|

Re: Where do the executors get my app jar from?

Henoc
If you are running Spark on Yarn, the spark-submit utility will download the jar from S3 and copy it to HDFS in a distributed cache. The driver shares this location with Yarn NodeManagers via the container LaunchContext. NodeManagers localize the jar and place it on container classpath before they launch the executor container

Henoc

On Fri, Aug 14, 2020, 6:19 AM Russell Spitzer <[hidden email]> wrote:
Looking back at the code

All --jar Args and such run through


Which calls 


Which places local jars on the driver hosted file server and just leaves Remote Jars as is with the path for executors to access them

On Thu, Aug 13, 2020 at 11:01 PM Russell Spitzer <[hidden email]> wrote:
The driver hosts a file server which the executors download the jar from.

On Thu, Aug 13, 2020, 5:33 PM James Yu <[hidden email]> wrote:
Hi,

When I spark submit a Spark app with my app jar located in S3, obviously the Driver will download the jar from the s3 location.  What is not clear to me is: where do the Executors get the jar from?  From the same s3 location, or somehow from the Driver, or they don't need the jar?

Thanks in advance for explanation.

James
Reply | Threaded
Open this post in threaded view
|

Re: Where do the executors get my app jar from?

Rishi Raj Tandon
Hi All,

What will happen in case the jar is available on the local network, which is accessible to Driver but not to executors.
Is there any good study resource where the deployment of exernal is explained nicely?

Regards,
Rishi 

On Fri, Aug 14, 2020 at 11:15 AM Henoc <[hidden email]> wrote:
If you are running Spark on Yarn, the spark-submit utility will download the jar from S3 and copy it to HDFS in a distributed cache. The driver shares this location with Yarn NodeManagers via the container LaunchContext. NodeManagers localize the jar and place it on container classpath before they launch the executor container

Henoc

On Fri, Aug 14, 2020, 6:19 AM Russell Spitzer <[hidden email]> wrote:
Looking back at the code

All --jar Args and such run through


Which calls 


Which places local jars on the driver hosted file server and just leaves Remote Jars as is with the path for executors to access them

On Thu, Aug 13, 2020 at 11:01 PM Russell Spitzer <[hidden email]> wrote:
The driver hosts a file server which the executors download the jar from.

On Thu, Aug 13, 2020, 5:33 PM James Yu <[hidden email]> wrote:
Hi,

When I spark submit a Spark app with my app jar located in S3, obviously the Driver will download the jar from the s3 location.  What is not clear to me is: where do the Executors get the jar from?  From the same s3 location, or somehow from the Driver, or they don't need the jar?

Thanks in advance for explanation.

James
Reply | Threaded
Open this post in threaded view
|

Re: Where do the executors get my app jar from?

James Yu
In reply to this post by Henoc
Henoc,

Ok. That is for Yarn with HDFS. What will happen in Kubernetes as resource manager without HDFS scenario? 

James


From: Henoc <[hidden email]>
Sent: Thursday, August 13, 2020 10:45 PM
To: James Yu <[hidden email]>
Cc: user <[hidden email]>; [hidden email] <[hidden email]>
Subject: Re: Where do the executors get my app jar from?
 
If you are running Spark on Yarn, the spark-submit utility will download the jar from S3 and copy it to HDFS in a distributed cache. The driver shares this location with Yarn NodeManagers via the container LaunchContext. NodeManagers localize the jar and place it on container classpath before they launch the executor container

Henoc

On Fri, Aug 14, 2020, 6:19 AM Russell Spitzer <[hidden email]> wrote:
Looking back at the code

All --jar Args and such run through


Which calls 


Which places local jars on the driver hosted file server and just leaves Remote Jars as is with the path for executors to access them

On Thu, Aug 13, 2020 at 11:01 PM Russell Spitzer <[hidden email]> wrote:
The driver hosts a file server which the executors download the jar from.

On Thu, Aug 13, 2020, 5:33 PM James Yu <[hidden email]> wrote:
Hi,

When I spark submit a Spark app with my app jar located in S3, obviously the Driver will download the jar from the s3 location.  What is not clear to me is: where do the Executors get the jar from?  From the same s3 location, or somehow from the Driver, or they don't need the jar?

Thanks in advance for explanation.

James