Single Leader Replication - Failover

 Single Leader Replication- Failover

As discussed in the last section (Replication), there are three popular algorithm for replication. One of them is single leader replication.
Single leader replication have only one leader and followers. 
Clients connect to only leader for write, however they can connect to followers for read.

We will understand more about failover in this section.

Follower failure : Achieve High Availability

  • On its local disk, each of the follower keeps a log of the data changes it has received from the leader.
  • If a follower crashes and is restarted or if the network between follower and leader is interrupted, the follower can recover quite easily from it's log.
  • Log tells the last transaction that was processed before the fault occurred. Thus the follower can connect to the leader and request all the data changes that occurred during the time when the follower was disconnected.
  • When it has applied the changes, it caught up to the leader and can continue receiving a stream of data changes as before.

Leader failure : Failover

  • If leader is failed, one of the follower to be promoted to the new leader.
  • Clients need to be reconfigured to send their writes to the new leader.
  • Other followers need to start consuming data changes from the new leader.
  • This complete process is called failover.

Automatic Failover Process :

  • First of all how to determine that the leader has failed
    • Time out technique
    • Nodes frequently bounce messages back and fourth between each other , if node does not respond for some period of time (say 30 secs), it is assumed to be dead.
    • Once it is confirmed that leader is failed, then its time to chose new leader
  • Choosing a new leader
    • This could be done through an election process.
    • The best candidate for leadership is usually the replica with the most up to date data changes from old leader.
    • Getting all the node to agree on a new leader is a consensus problem.
  • Reconfiguring
    • Once new leader is selected, now client needs to send write request to the new leader.
    • If the old leader comes back, it still believe that it is the leader. The system needs to ensure that the old leader becomes the follower and recognize the new leader.

Trade-off in failover : 

  • Asynchronous Replication: 
    • If Asynchronous replication is used, the new leader may not have up to date data.
    • If old leader rejoin to the cluster, what should happen to those writes (data which are up to date on older leader).
      • The new leader may have received conflicting writes in the meantime.
      • The most common solution is that old leader's un-replicated data to simply  be discarded, which will violate data durability expectation.
  • Split Brain:
    • In certain fault scenario, it could happen that two nodes both believe that they are leader. This situation is called split-brain.
    • This is dangerous, if both leaders accept writes and there is no process to resolving conflicts, data is likely to be lost or corrupted.
    • As a safety catch, some system have a mechanism to shut down one node if two leaders are detected.
    • However, if this mechanism is not carefully designed, you can end-up with both nodes being shut down.

What is the ideal time out before the leader is declared dead:

  • A longer timeout means a longer time to recovery in the case where the leader fails.
  • If timeout is too short, there could be unnecessary  failover. e,g, a temporary load spike could cause a node's response time to increase above the timeout or network glitch could cause delayed packets.
Most of the concepts explained here are common for leader based replication(single or Multi leaders)

Comments

Post a Comment

Popular posts from this blog

Distributed Lock with Redlock

 Distributed Lock with Redlock Distributed lock systems are used to ensure that only one process or thread can access a particular resource at a time. Redlock is a distributed lock system that was introduced by Redis. The goal of Redlock is to provide a reliable and fault-tolerant distributed lock service.   The Redlock system is designed to work in a distributed environment where multiple instances of a service or application can run concurrently. Each instance of the application can acquire a lock and perform some operations on the resource. The Redlock system ensures that only one instance of the application has access to the resource at a time.   Components of Redlock :   Lock Manager : The Lock Manager is responsible for managing the lock state and deciding which instance of the application can acquire the lock. It receives requests from the applications to acquire or release the lock.   The Lock Manager uses a distributed consensus algorithm to en...
Read more

Distributed Transaction

Distributed Transaction A distributed transaction across multiple database services involves multiple databases that are geographically distributed across  different locations .       Here are some important steps to consider when designing a system for distributed transactions across multiple database services:     Transaction Protocol      Transactional consistency      Concurrency control      Reliability and Fault-tolerance      Performance   Transactional Protocol A distributed transaction protocol is a set of rules and procedures that define how a distributed transaction should be executed across multiple databases .   It is used for Atomic commit across databases    There are several distributed transaction protocols to choose from, each with its own advantages and disadvantages.     Two-Phase Commit (2PC ) :   It is a widely-used protocol that ensures at...
Read more

Storage Engine

Image
Database Storage Engine Two kinds of database storage engine are most popular: Log Structured Storage Engine (SSTable, LSMTree) Page Oriented Storage Engine (B-Tree) Log Structured Storage Engine  Most of the databases internally use a log, which is an append only data file. Log file is an append only sequence of records It is not human readable, binary file Issues Since log file is append only file, so it keeps growing, hence resulting performance issue. While reading data, every time it looks from beginning Performance degrades if number of records are more in the database To mitigate these issues, a new concept came out called Indexing. Index In order to efficiently find the value for a particular key in the database, we need a different data structure, this is called an Index. In General the idea behind them is to keep some additional metadata on the side, which acts as a signpost and helps you to locate the data you want. If you want to search some data in a several different ...
Read more