Flow Balance

At the core of Happen's resilience capabilities is Flow Balance, a powerful yet invisible mechanism that leverages NATS features to monitor the health of your distributed system through natural message flow patterns.

Flow Balance builds on NATS JetStream's built-in monitoring capabilities to provide insights into your distributed system's health:

• Message Tracking: NATS JetStream tracks message delivery and acknowledgment

• Consumer Lag: Happens detects when consumers fall behind producers

• Delivery Failures: JetStream's acknowledgment system reveals delivery problems

• Partition Detection: Patterns in message flow reveal network partitions

This NATS-powered approach gives Happen a natural way to detect issues in your distributed system without requiring complex configuration or special APIs.

How Flow Balance Works

Flow Balance operates by monitoring the natural patterns in your message flow through NATS:

1. JetStream Consumer Monitoring: Happen leverages JetStream's consumer metrics to track delivery success

2. Lag Detection: When consumers fall behind producers, Happen detects potential issues

3. Event Emission: When issues are detected, Happen emits system events that applications can respond to

Unlike traditional health monitoring systems that rely on external probes, Flow Balance uses the message flow itself as an indicator of system health.

Observable Patterns

Different types of issues create distinctive flow patterns that your handlers can interpret:

1. Network Partitions:

   • Pattern: Sharp drop in delivery across node groups

   • Distribution: Clear grouping pattern - nodes on one side can't reach nodes on the other

   • Timing: Occurs suddenly and affects multiple nodes simultaneously

2. Node Failures:

   • Pattern: Delivery failures specifically for one node

   • Distribution: Concentrated around the failed node

   • Timing: Occurs suddenly for the specific node

3. Processing Bottlenecks:

   • Pattern: Gradually increasing consumer lag for a specific node

   • Distribution: Usually affects a single node or service

   • Timing: Develops gradually over time

4. System Overload:

   • Pattern: Rising consumer lag across most or all nodes

   • Distribution: Affects most components somewhat equally

   • Timing: Often correlates with increased event volume

By analyzing these patterns in your handlers, you can determine the specific type of issue occurring and implement appropriate recovery strategies.

Imbalance Events

When Flow Balance detects issues, it emits events that your application can listen for and respond to:

// Listen for node-specific imbalance events
happen.on("node.down", (event) => {
  const { nodeId, lagMetrics, pattern } = event.payload;
  
  // Implement your recovery strategy
  if (lagMetrics.messagesWaiting > 1000) {
    // Severe imbalance - potential failure
    implementNodeFailureRecovery(nodeId);
  } else if (lagMetrics.messagesWaiting > 500) {
    // Moderate imbalance - potential bottleneck
    applyBackpressure(nodeId);
  } else {
    // Minor imbalance - monitor
    logImbalance(nodeId, lagMetrics);
  }
});

// Listen for system-wide imbalance events
happen.on("system.down", (event) => {
  const { level, affectedNodes, pattern } = event.payload;
  
  // Implement system-wide recovery
  if (level === "critical") {
    // Severe system-wide issue
    enableEmergencyMode();
  } else if (level === "warning") {
    // Moderate system-wide issue
    throttleNonEssentialOperations();
  }
});

Building Recovery Strategies

Happen provides the detection capabilities through NATS, and you can implement recovery strategies based on your application's needs:

// Listen for potential partition events
happen.on("node.down", (event) => {
  const { nodeId, lagMetrics, affectedNodes } = event.payload;
  
  // Check for partition pattern
  if (isPotentialPartition(nodeId, affectedNodes, lagMetrics)) {
    // Implement partition recovery strategy
    enablePartitionMode({
      isolatedNodes: affectedNodes,
      prioritizeLocalOperations: true,
      queueRemoteOperations: true
    });
    
    // Notify operations team
    alertOperations({
      type: "network-partition",
      affectedNodes,
      detectedAt: Date.now()
    });
  }
});

Benefits of Flow Balance

This approach provides several key advantages:

1. Zero API Surface: No special methods or configuration parameters

2. Zero Additional Overhead: Uses existing NATS monitoring capabilities

3. Natural Detection: System issues reveal themselves through flow patterns

4. Customizable Response: You control how your application responds to different issues

5. Transport Independence: Works the same across all NATS deployment models

6. Observable Patterns: Clear indicators of different types of system issues

By leveraging NATS's built-in monitoring capabilities, Happen provides powerful system insights with minimal overhead.