Possible "split brain" situation

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Possible "split brain" situation

Gimantha Bandara
Hi all,

We are using embedded Spark 1.6.2 in our analytics platform[1]. For the cluster communication we use hazel-cast clustering capabilities. From Hazelcast side we set the following configurations, in order to configure the hearbeat properties.

hazelcast.max.no.heartbeat.seconds=30
hazelcast.max.no.master.confirmation.seconds=45

So, if there is a network failure for more than 30 seconds,  Hazelcast detects that this is a network outage. So each node sees that the other node left the cluster (in a 2 node cluster). What happens is, the current active node stays active and the passive master also becomes active because of the network outage. Following are the respective messages printed by hazelcast.

node 1:
TID: [-1] [] [2017-11-05 08:59:14,071]  INFO {org.wso2.carbon.core.clustering.hazelcast.wka.WKABasedMembershipScheme} -  Member left [ff9788ad-8570-465c-846a-82393c636ae5]: /10.36.239.70:4000 {org.wso2.carbon.core.clustering.hazelcast.wka.WKABasedMembershipScheme}

node 2:
TID: [-1] [] [2017-11-05 09:00:40,357]  INFO {org.wso2.carbon.core.clustering.hazelcast.wka.WKABasedMembershipScheme} -  Member left [d02e49f5-78ab-4bfb-ba2c-df24f7e5c058]: /10.36.239.67:4000 {org.wso2.carbon.core.clustering.hazelcast.wka.WKABasedMembershipScheme}


When the network recovers, we see the following error in Spark,

ERROR {org.apache.spark.deploy.worker.Worker} -  Worker registration failed: Duplicate worker ID {org.apache.spark.deploy.worker.Worker}


This also cause our server to shutdown(please note that we spawn a spark JVM from our server). Please see the following shutdown error. 

INFO {org.wso2.carbon.core.init.CarbonServerManager} -  Shutdown hook triggered.... {org.wso2.carbon.core.init.CarbonServerManager}

Do you guys have any idea what is happening to a Spark cluster when there is a network outage and how it handles the split brain situations? Is there a way to restore the previous cluster state, once both nodes join the cluster again? Also I am curious to know why the shudown hook is triggered.

Thanks in advance!
Gimantha