cannot access port 4040

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
mrm
Reply | Threaded
Open this post in threaded view
|

cannot access port 4040

mrm
This post has NOT been accepted by the mailing list yet.
Hi,

I am using Spark 1.3.1 standalone and I have a problem where my cluster is working fine, I can see the port 8080 and check that my ec2 instances are fine, but I cannot access port 4040.

I have tried sbin/stop-all.sh, sbin/stop-master.sh, exiting the spark context and restarting it to no avail.

Any clues on what to try next?

Thanks,
Maria
mrm
Reply | Threaded
Open this post in threaded view
|

Re: cannot access port 4040

mrm
This post has NOT been accepted by the mailing list yet.
Hi Akhil,

(Your reply does not appear in the mailing list but I received an email so I will reply here).

I have an application running already in the shell using pyspark. I can see the application running on port 8080, but I cannot log into it through port 4040. It says "connection timed out" after a while. I tried relaunching my cluster using the spark-ec2 script but still no success.
mrm
Reply | Threaded
Open this post in threaded view
|

Re: cannot access port 4040

mrm
This post has NOT been accepted by the mailing list yet.
Hi Akhil,

Thanks for your reply! I still cannot see port 4040 in my machine when I type "master-ip-address:4040" in my browser.

I have tried this command: netstat -nat | grep 4040 and it returns this:
tcp        0      0 :::4040                     :::*                        LISTEN

Logging into my master is not a problem since I can access port 8080 by writing "master-ip-address:8080" in my browser.

I have made sure that spark.ui.enabled was set to True by launching my application using: ~/spark/bin/pyspark --conf spark.ui.enabled=True.

I don't know if this is a symptom of the problem that I have, but it might be another piece of useful information. When I look at "Completed Applications" in port 8080, I see my two previous applications. One of them says "cores: 160", the last one has "cores: 0". Could this be a clue?



mrm
Reply | Threaded
Open this post in threaded view
|

Re: cannot access port 4040

mrm
This post has NOT been accepted by the mailing list yet.
Hi,

Akhil Das suggested using ssh tunnelling (ssh -L 4040:127.0.0.1:4040 master-ip, and then open localhost:4040 in browser.) and this solved my problem, so it made me think that the settings of my cluster were wrong.

So I checked the inbound rules for the security group of my cluster and I realised that the port 4040 was missing!

Now I can log back into port 4040.