Spark block manager registration extreme slow

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Spark block manager registration extreme slow

Denes
Hi,

My Spark installations (both 0.9.1 and 1.0.0) starts up extremely slow when starting a simple Spark Streaming job.

I have to wait 6 (!) minutes at

INFO storage.BlockManagerMasterActor$BlockManagerInfo: Registering block manager
stage and another 4 (!) minutes at
INFO util.MetadataCleaner: Ran metadata cleaner for DAG_SCHEDULER

This is a sum of 10 minute wait to start a job which is really unacceptable... Can someone help me out where to look for a solution?