Monitor the network Communication

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

Monitor the network Communication


   I try to monitor the spark network communication, not the total network communication in my node.

   So I need the communication port, I figured out that the ConnectionManager port in the spark is random assigned, I can get the port from the BlockManager in the master when every node register their BlockManager, and send the port to Ganglia, and then start monitor this port in every node.

   But I found this method is a little complicated, and I do not sure if this can work well, so is there any good method to monitor the spark network communication? thanks!

   btw: What is the network communication data of Ganglia run on EC2, the total network package on the node?