[spark-core] docker-image-tool.sh question...

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

[spark-core] docker-image-tool.sh question...

muthu
Hello there,

While using docker-image-tool (for Spark 3.1.1) it seems to not accept `java_image_tag` property. The docker image default to JRE 11. Here is what I am running from the command line.

$ spark/bin/docker-image-tool.sh -r docker.io/sample-spark -b java_image_tag=8-jre-slim -t 3.1.1 build

Please advice,
Muthu
Reply | Threaded
Open this post in threaded view
|

Re: [spark-core] docker-image-tool.sh question...

Attila Zsolt Piros
Hi Muthu!

I tried and at my side it is working just fine:

$  ./bin/docker-image-tool.sh -r docker.io/sample-spark -b java_image_tag=8-jre-slim -t 3.1.1 build
Sending build context to Docker daemon  228.3MB
Step 1/18 : ARG java_image_tag=11-jre-slim
Step 2/18 : FROM openjdk:${java_image_tag}
8-jre-slim: Pulling from library/openjdk
45b42c59be33: Pull complete
c3f1fbf102b7: Pull complete
262868e4544c: Pull complete
1c0fec43ba3f: Pull complete
Digest: sha256:412c52d88d77ea078c50ed4cf8d8656d6448b1c92829128e1c6aab6687ce0998
Status: Downloaded newer image for openjdk:8-jre-slim
 ---> 8f867fdbd02f

What you see at your side?

Best regards,
Attila

On Wed, Mar 10, 2021 at 5:44 AM Muthu Jayakumar <[hidden email]> wrote:
Hello there,

While using docker-image-tool (for Spark 3.1.1) it seems to not accept `java_image_tag` property. The docker image default to JRE 11. Here is what I am running from the command line.

$ spark/bin/docker-image-tool.sh -r docker.io/sample-spark -b java_image_tag=8-jre-slim -t 3.1.1 build

Please advice,
Muthu
Reply | Threaded
Open this post in threaded view
|

Re: [spark-core] docker-image-tool.sh question...

muthu
Hello Attila,

Thank you for verifying this for me. I was looking at
Step 1/18 : ARG java_image_tag=11-jre-slim
and presumed that the docker image is built using JRE 11.

I can confirm that,
(1) $ docker image history 3ef86250a35b
IMAGE          CREATED              CREATED BY                                      SIZE      COMMENT
3ef86250a35b   About a minute ago   /bin/sh -c #(nop)  USER 185                     0B
da69b1ceffae   About a minute ago   /bin/sh -c #(nop)  ENTRYPOINT ["/opt/entrypo…   0B
64a6705c5b0a   About a minute ago   |1 spark_uid=185 /bin/sh -c chmod a+x /opt/d…   1.28kB
31998314a897   About a minute ago   |1 spark_uid=185 /bin/sh -c chmod g+w /opt/s…   0B
80e5a5efb145   About a minute ago   /bin/sh -c #(nop) WORKDIR /opt/spark/work-dir   0B
0679f96869ca   About a minute ago   /bin/sh -c #(nop)  ENV SPARK_HOME=/opt/spark    0B
924e4145db72   About a minute ago   /bin/sh -c #(nop) COPY dir:bbafc09edfdb15f8a…   969kB
2d0c1f6457e1   About a minute ago   /bin/sh -c #(nop) COPY dir:5bfd4e5c910d2d60b…   11.3kB
22faf9d2953c   About a minute ago   /bin/sh -c #(nop) COPY dir:fe08a455f93331449…   3.05MB
86105d076b70   About a minute ago   /bin/sh -c #(nop) COPY file:35ddca176fca5de0…   1.28kB
79328c9a5f9f   About a minute ago   /bin/sh -c #(nop) COPY file:effc73fa9a595c48…   3.49kB
6691778d46d9   About a minute ago   /bin/sh -c #(nop) COPY dir:95444b482efa0f5f8…   45.1kB
6e9b0eb8dd36   About a minute ago   /bin/sh -c #(nop) COPY dir:a67a7095c35356a11…   53.8kB
a9d17543e0cb   About a minute ago   /bin/sh -c #(nop) COPY dir:087058b48f1aced38…   447MB
12ecbd1bb3bc   2 minutes ago        |1 spark_uid=185 /bin/sh -c set -ex &&     s…   79.1MB
1f4a72f317c2   2 months ago         /bin/sh -c #(nop)  ARG spark_uid=185            0B
cd08b38dfcae   2 months ago         /bin/sh -c set -eux;   arch="$(dpkg --print-…   109MB
<missing>      2 months ago         /bin/sh -c #(nop)  ENV JAVA_VERSION=8u275       0B
<missing>      2 months ago         /bin/sh -c { echo '#/bin/sh'; echo 'echo "$J…   27B
<missing>      2 months ago         /bin/sh -c #(nop)  ENV PATH=/usr/local/openj…   0B
<missing>      2 months ago         /bin/sh -c #(nop)  ENV JAVA_HOME=/usr/local/…   0B
<missing>      2 months ago         /bin/sh -c #(nop)  ENV LANG=C.UTF-8             0B
<missing>      2 months ago         /bin/sh -c set -eux;  apt-get update;  apt-g…   8.78MB
<missing>      3 months ago         /bin/sh -c #(nop)  CMD ["bash"]                 0B
<missing>      3 months ago         /bin/sh -c #(nop) ADD file:3a7bff4e139bcacc5…   69.2MB

(2) $  docker run --entrypoint "/usr/local/openjdk-8/bin/java" 3ef86250a35b '-version'
openjdk version "1.8.0_275"
OpenJDK Runtime Environment (build 1.8.0_275-b01)
OpenJDK 64-Bit Server VM (build 25.275-b01, mixed mode)

Thank you much!

Thanks,
Muthu

On Wed, Mar 10, 2021 at 11:11 PM Attila Zsolt Piros <[hidden email]> wrote:
Hi Muthu!

I tried and at my side it is working just fine:

$  ./bin/docker-image-tool.sh -r docker.io/sample-spark -b java_image_tag=8-jre-slim -t 3.1.1 build
Sending build context to Docker daemon  228.3MB
Step 1/18 : ARG java_image_tag=11-jre-slim
Step 2/18 : FROM openjdk:${java_image_tag}
8-jre-slim: Pulling from library/openjdk
45b42c59be33: Pull complete
c3f1fbf102b7: Pull complete
262868e4544c: Pull complete
1c0fec43ba3f: Pull complete
Digest: sha256:412c52d88d77ea078c50ed4cf8d8656d6448b1c92829128e1c6aab6687ce0998
Status: Downloaded newer image for openjdk:8-jre-slim
 ---> 8f867fdbd02f

What you see at your side?

Best regards,
Attila

On Wed, Mar 10, 2021 at 5:44 AM Muthu Jayakumar <[hidden email]> wrote:
Hello there,

While using docker-image-tool (for Spark 3.1.1) it seems to not accept `java_image_tag` property. The docker image default to JRE 11. Here is what I am running from the command line.

$ spark/bin/docker-image-tool.sh -r docker.io/sample-spark -b java_image_tag=8-jre-slim -t 3.1.1 build

Please advice,
Muthu