Unable to run spark in local mode

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Unable to run spark in local mode

kamatsuoka
When I try to run spark in local mode on my Mac, it get stuck trying to fetch my application jar.

My command line is:

SPARK_CLASSPATH=~/depot/Engineering/kenji/dadlog-filter/target/scala-2.10/dadlog-filter-assembly-1.0.jar ~/opt/spark/bin/spark-class {myMainClass} local s3n://{myInputFile}

Spark finds {myMainClass} but gets stuck here:

14/02/07 12:09:59 INFO Executor: Fetching http://10.16.5.171:64530/jars/dadlog-filter-assembly-1.0.jar with timestamp 1391803795910
14/02/07 12:09:59 INFO Utils: Fetching http://10.16.5.171:64530/jars/dadlog-filter-assembly-1.0.jar to /var/folders/94/kz0lxyl56dv6rz9znvrndydh0000gn/T/fetchFileTemp1339484021541958973.tmp

The temp file is 64k in size.  It's as though it reads the first 64k then hangs.