In distributed systems, often times a central coordination service is needed between nodes.

Apache ZooKeeper

Apache ZooKeeper is an example of a coordination service that can be used to coordinate nodes.

Use Case

NoSQL with Strong Consistency for Unique Id Generation

• This example stems from the URL Shortener system design, where a unique id needs to be created but ensured it's not in use
  • in SQL we are ACID compliant and can use putIfAbsent for Strong consistency (read after write)
  • in NoSQL, we are running on Eventual consistency due to the nature of how it scales horizontally
  • Checking if it exists each new post / put request is expensive and not viable
• We want to use NoSQL for the scalability, but can not have eventual consistency as it is a risk of having collision, this is how we can achieve NoSQL with Strong Consistency when generating ids:
  1. A number from a counter can be used to create random unique ids (using encoding or hashing) and ensure it has not been used before (since it increments)
  2. Lets say you have an application with many instances / nodes and is regionally distributed and we need to keep track of a counter 1 - 100,000,000 between all instances and be synchronized
  3. We can use ZooKeeper to coordinate instance 1 counts: 0 - 10,000,000 instance 2 counts: 10,000,001 - 20,000,000 ... and when an instance counter reaches the limit, the coordination service will tell the instance the next range to use and if an instance goes down, that range will simply be unused or paused