Unable to run spark in local mode

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

Unable to run spark in local mode

When I try to run spark in local mode on my Mac, it get stuck trying to fetch my application jar.

My command line is:

SPARK_CLASSPATH=~/depot/Engineering/kenji/dadlog-filter/target/scala-2.10/dadlog-filter-assembly-1.0.jar ~/opt/spark/bin/spark-class {myMainClass} local s3n://{myInputFile}

Spark finds {myMainClass} but gets stuck here:

14/02/07 12:09:59 INFO Executor: Fetching with timestamp 1391803795910
14/02/07 12:09:59 INFO Utils: Fetching to /var/folders/94/kz0lxyl56dv6rz9znvrndydh0000gn/T/fetchFileTemp1339484021541958973.tmp

The temp file is 64k in size.  It's as though it reads the first 64k then hangs.