Spark Training Scripts (from Strata 09/13) Ec2 Deployment scripts having errors

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Spark Training Scripts (from Strata 09/13) Ec2 Deployment scripts having errors

Stephen Boesch
The spark-training scripts are not  presently working 100%: the errors displayed when starting the slaves are shown below. 

Possibly a newer location for the files exists (I pulled from https://github.com/amplab/training-scripts an it is nearly 6 months old)


cp: cannot create regular file `/root/mesos-ec2/': Is a directory


cp: cannot create regular file `/root/mesos-ec2/cluster-url': No such file or directory
rsync: link_stat "/root/mesos-ec2" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1039) [sender=3.0.6]
rsync: link_stat "/root/mesos-ec2" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1039) [sender=3.0.6]
..
main.c(1039) [sender=3.0.6]
./spark-standalone/setup.sh: line 18: /root/spark/bin/stop-all.sh: No such file or directory
./spark-standalone/setup.sh: line 23: /root/spark/bin/start-master.sh: No such file or directory
./spark-standalone/setup.sh: line 29: /root/spark/bin/start-slaves.sh: No such file or directory

~ ~/spark-ec2
./training/setup.sh: line 36: pushd: /root/MLI: No such file or directory
fatal: Not a git repository (or any of the parent directories): .git
fatal: Not a git repository (or any of the parent directories): .git
fatal: Not a git repository (or any of the parent directories): .git
./training/setup.sh: line 41: ./sbt/sbt: No such file or directory
RSYNC'ing /root/MLI to slaves...
rsync: link_stat "/root/MLI" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1039) [sender=3.0.6]
rsync: link_stat "/root/MLI" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1039) [sender=3.0.6]
..
main.c(1039) [sender=3.0.6]
~/spark-ec2
./training/setup.sh: line 46: pushd: /root/hive_blinkdb: No such file or directory
Already up-to-date.
Buildfile: build.xml does not exist!
Build failed
RSYNC'ing /root/hive_blinkdb to slaves...
rsync: link_stat "/root/hive_blinkdb" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1039) [sender=3.0.6]
..
main.c(1039) [sender=3.0.6]
./training/setup.sh: line 50: popd: directory stack empty
~/blinkdb ~/spark-ec2

Waiting for cluster to start...
Traceback (most recent call last):
  File "./spark_ec2.py", line 915, in <module>
    main()
  File "./spark_ec2.py", line 758, in main
    err = wait_for_spark_cluster(master_nodes, opts)
  File "./spark_ec2.py", line 724, in wait_for_spark_cluster
    err = check_spark_cluster(master_nodes, opts)
  File "./spark_ec2.py", line 453, in check_spark_cluster
    response = urllib2.urlopen(url)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 127, in urlopen
    return _opener.open(url, data, timeout)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 404, in open
    response = self._open(req, data)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 422, in _open
    '_open', req)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 382, in _call_chain
    result = func(*args)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1214, in http_open
    return self.do_open(httplib.HTTPConnection, req)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1184, in do_open
    raise URLError(err)
urllib2.URLError: <urlopen error [Errno 61] Connection refused>
Reply | Threaded
Open this post in threaded view
|

Re: Spark Training Scripts (from Strata 09/13) Ec2 Deployment scripts having errors

Stephen Boesch
I found updated materials (ampcamp4 from Feb 2014)  and tried the instructions at http://ampcamp.berkeley.edu/4/exercises/launching-a-bdas-cluster-on-ec2.html.

We have some similar errors even on the latest

cp: cannot create regular file `/root/mesos-ec2/': Is a directory


RSYNC'ing /root/mesos-ec2 to slaves...
rsync: link_stat "/root/mesos-ec2" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1039) [sender=3.0.6]
rsync: link_stat "/root/mesos-ec2" failed: No such file or directory (2)

But  in this latest version the mesos errors appear not to be  fatal: the cluster is in the process of coming up (copying wikipedia data now..)
.



2014-03-08 6:26 GMT-08:00 Stephen Boesch <[hidden email]>:
The spark-training scripts are not  presently working 100%: the errors displayed when starting the slaves are shown below. 

Possibly a newer location for the files exists (I pulled from https://github.com/amplab/training-scripts an it is nearly 6 months old)


cp: cannot create regular file `/root/mesos-ec2/': Is a directory


cp: cannot create regular file `/root/mesos-ec2/cluster-url': No such file or directory
rsync: link_stat "/root/mesos-ec2" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1039) [sender=3.0.6]
rsync: link_stat "/root/mesos-ec2" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1039) [sender=3.0.6]
..
main.c(1039) [sender=3.0.6]
./spark-standalone/setup.sh: line 18: /root/spark/bin/stop-all.sh: No such file or directory
./spark-standalone/setup.sh: line 23: /root/spark/bin/start-master.sh: No such file or directory
./spark-standalone/setup.sh: line 29: /root/spark/bin/start-slaves.sh: No such file or directory

~ ~/spark-ec2
./training/setup.sh: line 36: pushd: /root/MLI: No such file or directory
fatal: Not a git repository (or any of the parent directories): .git
fatal: Not a git repository (or any of the parent directories): .git
fatal: Not a git repository (or any of the parent directories): .git
./training/setup.sh: line 41: ./sbt/sbt: No such file or directory
RSYNC'ing /root/MLI to slaves...
rsync: link_stat "/root/MLI" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1039) [sender=3.0.6]
rsync: link_stat "/root/MLI" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1039) [sender=3.0.6]
..
main.c(1039) [sender=3.0.6]
~/spark-ec2
./training/setup.sh: line 46: pushd: /root/hive_blinkdb: No such file or directory
Already up-to-date.
Buildfile: build.xml does not exist!
Build failed
RSYNC'ing /root/hive_blinkdb to slaves...
rsync: link_stat "/root/hive_blinkdb" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1039) [sender=3.0.6]
..
main.c(1039) [sender=3.0.6]
./training/setup.sh: line 50: popd: directory stack empty
~/blinkdb ~/spark-ec2

Waiting for cluster to start...
Traceback (most recent call last):
  File "./spark_ec2.py", line 915, in <module>
    main()
  File "./spark_ec2.py", line 758, in main
    err = wait_for_spark_cluster(master_nodes, opts)
  File "./spark_ec2.py", line 724, in wait_for_spark_cluster
    err = check_spark_cluster(master_nodes, opts)
  File "./spark_ec2.py", line 453, in check_spark_cluster
    response = urllib2.urlopen(url)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 127, in urlopen
    return _opener.open(url, data, timeout)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 404, in open
    response = self._open(req, data)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 422, in _open
    '_open', req)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 382, in _call_chain
    result = func(*args)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1214, in http_open
    return self.do_open(httplib.HTTPConnection, req)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1184, in do_open
    raise URLError(err)
urllib2.URLError: <urlopen error [Errno 61] Connection refused>