Executor not getting added SparkUI & Spark Eventlog in deploymode:cluster

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

Executor not getting added SparkUI & Spark Eventlog in deploymode:cluster

Mamillapalli, Purna Pradeep

Hi all,


Im performing spark submit using Spark rest api POST operation on 6066 port with below config


> Launch Command:

> "/usr/lib/jvm/java-1.8.0-openjdk-"

> "-cp" "/usr/local/spark/conf/:/usr/local/spark/jars/*" "-Xmx4096M"

> "-Dspark.eventLog.enabled=true"

> "-Dspark.app.name=WorkflowApp"

> "-Dspark.submit.deployMode=cluster"

> "-Dspark.local.dir=/data0,/data1,/data2,/data3"

> "-Dspark.executor.cores=2" "-Dspark.master=spark://****:7077"

> "-Dspark.serializer=org.apache.spark.serializer.KryoSerializer"

> "-Dspark.jars=s3a://<***>.jar" "-Dspark.driver.supervise=false"

> "-Dspark.history.fs.logDirectory=s3a://<*>/"

> "-Dspark.hadoop.fs.s3a.server-side-encryption-algorithm=AES256"

> "-Dspark.driver.memory=4G" "-Dspark.executor.memory=4G"

> "-Dspark.eventLog.dir=s3a://<*>/"

> "org.apache.spark.deploy.worker.DriverWrapper" "spark://Worker@<***>"

> "/usr/local/spark/work/driver-<***>.jar" "MyApp" "-c" "s3a://<***>"



when i looked into Spark eventlog below is what i observed


    {"Event":"SparkListenerExecutorAdded","Timestamp":1510633498623,"Executor ID":"driver","Executor Info":{"Host":"localhost","Total Cores":2,"Log Urls":{}}}




Though i ran in deployMode as cluster  the slave ip is not shown in Host section & spark.master is shown as local[*] above ,because of this the job is running only on driver and therefore when job is submitted its not showing up in http://<masterip>:8080  under Running and Completed applications and it shows only under Running Drivers & Completed Drivers. Please suggest


The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.