problem starting the history server on EC2

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

problem starting the history server on EC2

zhen
I created a Spark 1.0 cluster on EC2 using the provided scripts. However, I do not seem to be able to start the history server on the master node. I used the following command:

./start-history-server.sh /root/spark_log


The error message says that the logging directory /root/spark_log does not exist. But I have definitely created the directory and made sure everyone can read/write/execute in the directory.

Can you tell me why it  does not work?

Thank you

Zhen
Reply | Threaded
Open this post in threaded view
|

Re: problem starting the history server on EC2

bc Wong

What's the permission on /root itself?

On Jun 10, 2014 6:29 PM, "zhen" <[hidden email]> wrote:
I created a Spark 1.0 cluster on EC2 using the provided scripts. However, I
do not seem to be able to start the history server on the master node. I
used the following command:

./start-history-server.sh /root/spark_log


The error message says that the logging directory /root/spark_log does not
exist. But I have definitely created the directory and made sure everyone
can read/write/execute in the directory.

Can you tell me why it  does not work?

Thank you

Zhen



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/problem-starting-the-history-server-on-EC2-tp7361.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
Reply | Threaded
Open this post in threaded view
|

Re: problem starting the history server on EC2

zhen
I checked the permission on root and it is the following:

drwxr-xr-x 20 root root  4096 Jun 11 01:05 root

So anyway, I changed to use /tmp/spark_log instead and this time I made sure that all permissions are given to /tmp and /tmp/spark_log like below. But it still does not work:

drwxrwxrwt  8 root root  4096 Jun 11 02:08 tmp
drwxrwxrwx 2 root root   4096 Jun 11 02:08 spark_log

Thanks

Zhen
Reply | Threaded
Open this post in threaded view
|

Re: problem starting the history server on EC2

Andrew Or-2
Can you try file:/root/spark_log?


2014-06-10 19:22 GMT-07:00 zhen <[hidden email]>:
I checked the permission on root and it is the following:

drwxr-xr-x 20 root root  4096 Jun 11 01:05 root

So anyway, I changed to use /tmp/spark_log instead and this time I made sure
that all permissions are given to /tmp and /tmp/spark_log like below. But it
still does not work:

drwxrwxrwt  8 root root  4096 Jun 11 02:08 tmp
drwxrwxrwx 2 root root   4096 Jun 11 02:08 spark_log

Thanks

Zhen



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/problem-starting-the-history-server-on-EC2-tp7361p7370.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

Reply | Threaded
Open this post in threaded view
|

Re: problem starting the history server on EC2

zhen
Sure here it is:

drwxrwxrwx  2 1000 root 4096 Jun 11 01:05 spark_logs

Zhen
Reply | Threaded
Open this post in threaded view
|

Re: problem starting the history server on EC2

Andrew Or-2
No, I meant pass the path to the history server start script.


2014-06-10 19:33 GMT-07:00 zhen <[hidden email]>:
Sure here it is:

drwxrwxrwx  2 1000 root 4096 Jun 11 01:05 spark_logs

Zhen



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/problem-starting-the-history-server-on-EC2-tp7361p7373.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

Reply | Threaded
Open this post in threaded view
|

Re: problem starting the history server on EC2

Krishna Sankar
In reply to this post by zhen
Yep, it gives tons of errors. I was able to make it work with sudo. Looks like ownership issue.
Cheers
<k/>


On Tue, Jun 10, 2014 at 6:29 PM, zhen <[hidden email]> wrote:
I created a Spark 1.0 cluster on EC2 using the provided scripts. However, I
do not seem to be able to start the history server on the master node. I
used the following command:

./start-history-server.sh /root/spark_log


The error message says that the logging directory /root/spark_log does not
exist. But I have definitely created the directory and made sure everyone
can read/write/execute in the directory.

Can you tell me why it  does not work?

Thank you

Zhen



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/problem-starting-the-history-server-on-EC2-tp7361.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

Reply | Threaded
Open this post in threaded view
|

Re: problem starting the history server on EC2

zhen
I tried everything including sudo, but it still did not work using the local directory.
However, I finally got it working by getting the history server to log into hdfs.
I first created a directory in hdfs like the following:
./ephemeral-hdfs/bin/hadoop fs -mkdir /spark_logs
Then I started the started the history server like the following.

./start-history-server.sh hdfs:///spark_logs --port 18080

In order to see the history server UI I needed to open up inbound traffic for the port 18080 in AWS. As follows

custom TCP  port 18080 from anywhere

Hope this will help others.

Zhen