Query on entrypoint.sh Kubernetes spark

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Query on entrypoint.sh Kubernetes spark

Sachit Murarka
Hi All,

To run spark on kubernetes . I see following lines in entrypoint.sh script available 

case "$1" in
  driver)
    shift 1
    CMD=(
      "$SPARK_HOME/bin/spark-submit"
      --conf "spark.driver.bindAddress=$SPARK_DRIVER_BIND_ADDRESS"
      --deploy-mode client

Could you pls suggest Why deploy-mode client is mentioned in entrypoint.sh ?
I am running spark submit using deploy mode cluster but inside entrypoint.sh which it is mentioned like that.


Kind Regards,
Sachit Murarka
Reply | Threaded
Open this post in threaded view
|

Re: Query on entrypoint.sh Kubernetes spark

Jacek Laskowski
Hi,

I'm a beginner in Spark on Kubernetes so bear with me and watch out for possible mistakes :)

The key to understand entrypoint.sh and -deploy-mode client is to think about the environment where the script is executed in. That's k8s already where the Docker image is brought to life as a container of a driver pod. There's no point using cluster deploy mode...ever. Makes sense?

On Thu, Jan 21, 2021 at 10:40 AM Sachit Murarka <[hidden email]> wrote:
Hi All,

To run spark on kubernetes . I see following lines in entrypoint.sh script available 

case "$1" in
  driver)
    shift 1
    CMD=(
      "$SPARK_HOME/bin/spark-submit"
      --conf "spark.driver.bindAddress=$SPARK_DRIVER_BIND_ADDRESS"
      --deploy-mode client

Could you pls suggest Why deploy-mode client is mentioned in entrypoint.sh ?
I am running spark submit using deploy mode cluster but inside entrypoint.sh which it is mentioned like that.


Kind Regards,
Sachit Murarka