This is a follow-up tutorial on the DRBD replication posts about resynchronization of sites after the failed PR site is recovered (DRBD split brain recovery).
More than a year ago I described in a long tutorial how to use DRBD in a real production environment and the challenges associated to it. See: DRBD based disk replication of a production cluster to a remote cluster on RHEL6
As a result of the DR promotion as the main site as instructed in DRBD: Promoting the DR site as the main site when PR is failed, after the PR site is recovered the so-called “split brain” condition will be experienced by the replication mechanism.
DRBD detects split brain at the time connectivity between PR and DR sites becomes available again and the nodes exchange the initial DRBD protocol handshake. If DRBD detects that both nodes are (or were at some point, while disconnected) in the primary role, it immediately tears down the replication connection. The tell-tale sign of this is a message like the following appearing in the system log:
Split-Brain detected, dropping connection!
After split brain has been detected, one node will always have the resource in a StandAlone connection state. The other might either also be in the StandAlone state (if both nodes detected the split brain simultaneously), or in WFConnection (if the peer tore down the connection before the other node had a chance to detect split brain).
We are selecting PR site as the node whose modifications will be discarded (this node is referred to as the split brain victim). This is because the split brain usually occurs after we already moved the main operation site to the DR site.
The split brain victim needs to be in the connection state of StandAlone or the following commands will return an error. You can ensure it is standalone by forcing disconnection of the repdata on the PR site:
# drbdadm disconnect repdata
On the PR site execute the following command to force the site as the secondary replication site (the site receiving the replicated data)
# drbdadm secondary repdata
On the PR site execute the following command to force the discarding of dirty data from the PR site. This dirty data is the data that was not replicated to the DR site and for which we do not have the certainty that is correct and not corrupted.
# drbdadm connect --discard-my-data repdata
On the DR site (the split brain survivor), if its connection state is also StandAlone, you would execute:
# drbdadm connect repdata
The above command will force the master drbd from the DR site to reinitialize the replication to the slave drbd from PR site. You may omit this step if the node is already in the WFConnection state; it will then reconnect automatically.
Upon connection, your split brain victim (PR site) immediately changes its connection state to SyncTarget, and has its modifications overwritten by the remaining primary node. After re-synchronization has completed, the split brain is considered resolved and the two nodes form a fully consistent, redundant replicated storage system again.
At any time the replication status can be easily monitored by running the following command on both sites:
#service drbd status
Note: After all these steps are done the DR site will still remain as the main operation site with the PR site as the secondary site. At any time the switch can be done as instructed in DRBD: Gracious role switching of live sites configured with DRBD replication but with the change that we promote PR site as the main site.
[paypal_donation_button]