Why NPE happen with multi threading in cluster mode but not client model

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Why NPE happen with multi threading in cluster mode but not client model

lk_spark
hi,all :
   I'm using spark2.4, I try to use multi thread to use sparkcontext , I found a example : https://hadoopist.wordpress.com/2017/02/03/how-to-use-threads-in-spark-job-to-achieve-parallel-read-and-writes/
   some code like this :
   for (a <- 0 until 4) {
      val thread = new Thread {
        override def run() {
          sc.parallelize(Array("ddd", "eee", "fff")).toDF().write.format("parquet").mode("overWrite").save("/tmp/vgiri/file2")
   .....
   
   when I run the code in local or client model , it will be work. But In cluster model I meet the ERROR: SPARK-29046 https://issues.apache.org/jira/browse/SPARK-29046
   finally , I use java.util.concurrent.Semaphore to wait all the sub thread to be finished before the main thread stop the sparkcontext , then the code worked in cluster model.

   But I don't understand why In local or client model ,even I didn't use a Semaphore, the code can also work ?