Quorum-Based Consensus

Coordinate concurrent operations safely across your cluster. Prevent race conditions and ensure only one critical operation runs at a time.

Real-World Use Case: Database Migration

You're migrating a database schema across 10 application servers. Only ONE server should run the migration at a time. Quorum-based locking ensures this.

1. Quorum Consensus

Locking API
# Acquire lock (requires majority agreement)
$ curl -X POST /api/v1.0/lock \
  -d '{
  "scope": "db-migration",
  "action": "lock"
}'

{
  "status": "locked",
  "scope": "db-migration",
  "quorum": "6/10 nodes agreed",
  "holder": "migration-orchestration-001"
}

# Release lock
$ curl -X POST /api/v1.0/lock \
  -d '{
  "scope": "db-migration",
  "action": "unlock"
}'

How It Works

  • Majority of nodes must agree (quorum)
  • Prevents split-brain scenarios
  • Automatic release on timeout
  • No single point of failure

Unlike Kubernetes' single etcd master, this is truly distributed.

2. DShell CLI - Interactive Lock Management

Operational Control via DShell

Interactive shell for managing locks without writing code:

  • manager lock acquire --scope SCOPE
  • manager lock status --scope SCOPE
  • manager lock release --scope SCOPE
  • Useful for operational teams and debugging

DShell provides a clean interface for HTTP APIs. Perfect for ops teams who prefer CLI over REST.

DShell Locking Commands
# Start interactive dshell
$ dshell -s cluster-node-1 -p 20194 -u admin

# Acquire distributed lock
dshell> manager lock acquire --scope db-migration
Lock acquired. Holder: dshell-session-001

# Check lock status
dshell> manager lock status --scope db-migration
LOCKED by dshell-session-001 (5/7 nodes)

# Release lock
dshell> manager lock release --scope db-migration
Lock released

# Batch mode for scripts
$ dshell manager lock acquire --scope db-migration

3. Orchestration Integration

Orchestration with Locking
{
  "name": "safe-db-migration",
  "steps": [
    {
      "name": "acquire-lock",
      "action_type": "http",
      "method": "POST",
      "url": "https://node/api/v1.0/lock",
      "body": {
        "scope": "db-migration",
        "action": "lock"
      }
    },
    {
      "name": "run-migration",
      "action_type": "shell",
      "code": "alembic upgrade head"
    },
    {
      "name": "release-lock",
      "action_type": "http",
      "method": "POST",
      "url": "https://node/api/v1.0/lock",
      "body": {
        "scope": "db-migration",
        "action": "unlock"
      }
    }
  ]
}

Lock within Orchestrations

Use locks to coordinate multi-step operations:

  • Acquire lock at step start
  • Run critical operations
  • Release lock on completion
  • Auto-release on failure
🔐

Safe Concurrency

Prevent race conditions without a central coordinator.

🤝

Consensus-Based

Quorum agreement ensures correctness even if nodes fail.

⏱️

Timeout Protection

Locks auto-release if holder crashes.

📍

Scope-Based

Multiple independent locks for different resources.