SparkContext taking time after adding jars and asking yarn for resources
In my production setup spark is always taking 40 seconds between these steps
like a fixed counter is set. In my local lab these steps take exact 1
second. I am not able to find the exact root cause of this behaviour. My
Spark application is running on Hortonworks platform in yarn client mode.
Can someone guide me what is happening between these steps
18/05/04 *07:56:45* INFO spark.SparkContext: Added JAR
file:/app/jobs/jobs.jar at spark://10.233.69.5:37668/jars/jobs.jar with
18/05/04 *07:57:26* WARN shortcircuit.DomainSocketFactory: The short-circuit
local reads feature cannot be used because libhadoop cannot be loaded.
18/05/04 *07:58:12* INFO client.AHSProxy: Connecting to Application History
server at gsidev001-mgt-01.thales.fr/192.168.1.11:10200