Testing Distributed Systems: Beyond Unit Tests

After implementing the transactional outbox pattern for the flight booking system, I had a moment of realization: how do I actually know it works? Sure, I can write unit tests for individual functions, but how do I test that events flow through the system correctly? That duplicate events are handled idempotently? That the system recovers when the message queue goes down?

This led me down the path of learning how to test distributed systems end-to-end, locally, without deploying to AWS. Here’s what I figured out.

The Testing Challenge

When you have a distributed system with:

Flight Service (writes to database + outbox)
Event Processor (reads outbox, publishes to queue)
Payment Service (consumes from queue, processes payments)
Message Queue (AWS SQS in production)
Database (PostgreSQL)

Unit tests only get you so far. You need to test the interactions between these components, and you need to test failure scenarios like:

What if the message queue is temporarily down?
What if the event processor crashes mid-processing?
What if events are delivered twice?
What if two services try to process the same event simultaneously?

The Local Testing Stack

To test this locally, I needed to run real infrastructure. Here’s the stack I set up:

Docker Compose - Orchestrates all services
PostgreSQL - Real database (not mocked)
LocalStack - Mocks AWS services (SQS) locally
Jest - Test framework for writing E2E tests

The key insight: don’t mock the infrastructure, run real versions locally. This gives you much higher confidence that your code will work in production.

Project Structure

flight-booking-system/
├── services/
│   ├── flight-service/       # Handles bookings, writes to outbox
│   ├── payment-service/      # Processes payments from queue
├── tests/
│   ├── e2e/                  # End-to-end tests
│   └── helpers/              # Test utilities
├── docker-compose.yml        # Local infrastructure setup
└── init-db.sql              # Database schema

Setting Up Local Infrastructure

Docker Compose Configuration

I created a docker-compose.yml that spins up everything needed:

version: '3.8'
 
services:
  postgres:
    image: postgres:15
    environment:
      POSTGRES_DB: flightdb
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: password
    ports:
      - "5432:5432"
    volumes:
      - ./init-db.sql:/docker-entrypoint-initdb.d/init-db.sql
 
  localstack:
    image: localstack/localstack:latest
    environment:
      SERVICES: sqs
    ports:
      - "4566:4566"
    volumes:
      - "./localstack-init:/etc/localstack/init/ready.d"
 
  flight-service:
    build: ./services/flight-service
    environment:
      DATABASE_URL: postgresql://postgres:password@postgres:5432/flightdb
      SQS_ENDPOINT: http://localstack:4566
      QUEUE_URL: http://localstack:4566/000000000000/flight-events.fifo
    ports:
      - "3000:3000"
    depends_on:
      - postgres
      - localstack
 
  payment-service:
    build: ./services/payment-service
    environment:
      DATABASE_URL: postgresql://postgres:password@postgres:5432/flightdb
      SQS_ENDPOINT: http://localstack:4566
      QUEUE_URL: http://localstack:4566/000000000000/flight-events.fifo
    depends_on:
      - postgres
      - localstack

Database Schema

The schema includes all the tables needed for the transactional outbox pattern:

-- Flights table
CREATE TABLE flights (
    id VARCHAR(255) PRIMARY KEY,
    passenger VARCHAR(255) NOT NULL,
    destination VARCHAR(255) NOT NULL,
    price INTEGER NOT NULL,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);
 
-- Outbox table (for reliable event publishing)
CREATE TABLE outbox (
    id SERIAL PRIMARY KEY,
    event_id VARCHAR(255) UNIQUE NOT NULL,
    aggregate_id VARCHAR(255) NOT NULL,
    event_type VARCHAR(255) NOT NULL,
    payload JSONB NOT NULL,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    processed_at TIMESTAMP
);
 
-- Payments table
CREATE TABLE payments (
    id SERIAL PRIMARY KEY,
    flight_id VARCHAR(255) UNIQUE NOT NULL,
    passenger VARCHAR(255) NOT NULL,
    amount INTEGER NOT NULL,
    status VARCHAR(50) NOT NULL,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);
 
-- Processed events table (for idempotency)
CREATE TABLE processed_events (
    event_id VARCHAR(255) PRIMARY KEY,
    processed_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

LocalStack Setup

LocalStack requires a small initialization script to create the SQS queue:

#!/bin/bash
awslocal sqs create-queue --queue-name flight-events.fifo \
  --attributes FifoQueue=true,ContentBasedDeduplication=false

Core Implementation Concepts

Flight Service - Transactional Outbox

The flight service writes to both the flights table and outbox table in a single transaction:

async function bookFlight(flight: Flight): Promise<void> {
  const client = await pool.connect();
  
  try {
    await client.query('BEGIN');
 
    // Insert flight
    await client.query(
      'INSERT INTO flights (id, passenger, destination, price) VALUES ($1, $2, $3, $4)',
      [flight.id, flight.passenger, flight.destination, flight.price]
    );
 
    // Insert outbox event (same transaction!)
    await client.query(
      'INSERT INTO outbox (event_id, aggregate_id, event_type, payload) VALUES ($1, $2, $3, $4)',
      [uuid(), flight.id, 'flight.booked', JSON.stringify(flight)]
    );
 
    await client.query('COMMIT');
  } catch (error) {
    await client.query('ROLLBACK');
    throw error;
  } finally {
    client.release();
  }
}

Event Processor - Polling and Publishing

A background process polls the outbox table and publishes events to SQS:

async function processOutbox(): Promise<void> {
  const events = await db.getUnprocessedEvents(10);
 
  for (const event of events) {
    try {
      // Publish to SQS
      await sqsClient.send(new SendMessageCommand({
        QueueUrl: QUEUE_URL,
        MessageBody: JSON.stringify({
          eventId: event.eventId,
          eventType: event.eventType,
          payload: event.payload
        }),
        MessageGroupId: event.aggregateId,
        MessageDeduplicationId: event.eventId
      }));
 
      // Mark as processed
      await db.markEventProcessed(event.eventId);
    } catch (error) {
      console.error('Failed to process event:', error);
      // Event remains unprocessed, will retry
    }
  }
}

Payment Service - Idempotent Processing

The payment service checks for duplicate events before processing:

async function processPayment(eventId: string, payment: Payment): Promise<void> {
  const client = await pool.connect();
 
  try {
    await client.query('BEGIN');
 
    // Check if already processed (within transaction)
    const alreadyProcessed = await client.query(
      'SELECT 1 FROM processed_events WHERE event_id = $1',
      [eventId]
    );
 
    if (alreadyProcessed.rows.length > 0) {
      console.log('Event already processed, skipping');
      await client.query('ROLLBACK');
      return;
    }
 
    // Process payment
    await client.query(
      'INSERT INTO payments (flight_id, passenger, amount, status) VALUES ($1, $2, $3, $4)',
      [payment.flightId, payment.passenger, payment.amount, 'completed']
    );
 
    // Mark event as processed
    await client.query(
      'INSERT INTO processed_events (event_id) VALUES ($1)',
      [eventId]
    );
 
    await client.query('COMMIT');
  } catch (error) {
    await client.query('ROLLBACK');
    throw error;
  } finally {
    client.release();
  }
}

Writing E2E Tests

The waitFor Pattern

Distributed systems are eventually consistent. Tests need to poll for conditions:

async function waitFor(
  condition: () => Promise<boolean>,
  options: { timeout?: number; interval?: number } = {}
): Promise<void> {
  const timeout = options.timeout || 10000;
  const interval = options.interval || 100;
  const start = Date.now();
 
  while (Date.now() - start < timeout) {
    if (await condition()) {
      return;
    }
    await new Promise(resolve => setTimeout(resolve, interval));
  }
 
  throw new Error('Condition not met within timeout');
}

Test Structure

A typical E2E test looks like this:

it('should book flight and process payment end-to-end', async () => {
  // Act: Book a flight
  await axios.post('http://localhost:3000/api/flights', {
    id: 'FL123',
    passenger: 'John Doe',
    destination: 'New York',
    price: 500
  });
 
  // Assert: Wait for payment to be processed
  await waitFor(async () => {
    const payments = await testDb.getPayments();
    return payments.length > 0;
  }, { timeout: 10000 });
 
  // Verify payment was created correctly
  const payments = await testDb.getPayments();
  expect(payments).toHaveLength(1);
  expect(payments[0].flight_id).toBe('FL123');
  expect(payments[0].amount).toBe(500);
});

Testing Idempotency

To test idempotency, manually inject a duplicate event:

it('should handle duplicate events idempotently', async () => {
  // Book a flight (creates event)
  await bookFlight({ id: 'FL123', passenger: 'John', price: 500 });
 
  // Wait for first processing
  await waitFor(async () => {
    const payments = await testDb.getPayments();
    return payments.length > 0;
  });
 
  // Get the event ID that was processed
  const processedEvents = await testDb.getProcessedEvents();
  const eventId = processedEvents[0].event_id;
 
  // Manually send duplicate event to SQS
  await sqsClient.send(new SendMessageCommand({
    QueueUrl: QUEUE_URL,
    MessageBody: JSON.stringify({
      eventId: eventId, // Same event ID!
      eventType: 'flight.booked',
      payload: { id: 'FL123', price: 500 }
    }),
    MessageGroupId: 'FL123',
    MessageDeduplicationId: `${eventId}-duplicate`
  }));
 
  // Wait for potential duplicate processing
  await new Promise(resolve => setTimeout(resolve, 3000));
 
  // Should still only have one payment
  const payments = await testDb.getPayments();
  expect(payments).toHaveLength(1);
});

Testing Failure Scenarios

Test what happens when services crash and recover:

it('should recover when payment service is temporarily down', async () => {
  // Book a flight
  await bookFlight({ id: 'FL123', price: 500 });
 
  // Stop payment service
  await execAsync('docker-compose stop payment-service');
 
  // Wait a bit - payment should NOT be processed
  await new Promise(resolve => setTimeout(resolve, 2000));
  let payments = await testDb.getPayments();
  expect(payments).toHaveLength(0);
 
  // Restart payment service
  await execAsync('docker-compose start payment-service');
  await new Promise(resolve => setTimeout(resolve, 3000));
 
  // Now payment should be processed
  await waitFor(async () => {
    payments = await testDb.getPayments();
    return payments.length > 0;
  });
 
  expect(payments[0].flight_id).toBe('FL123');
});

Running the Tests

The workflow I settled on:

# Start infrastructure
docker-compose up -d postgres localstack
 
# Start services
docker-compose up -d flight-service payment-service
 
# Wait for services to be ready
sleep 3
 
# Run E2E tests
npm run test:e2e
 
# Or run everything at once
npm test

The package.json includes scripts for each step:

{
  "scripts": {
    "test:setup": "docker-compose up -d postgres localstack && sleep 5",
    "test:services": "docker-compose up -d flight-service payment-service",
    "test:e2e": "jest --config jest.config.js --runInBand",
    "test:teardown": "docker-compose down -v",
    "test": "npm run test:setup && npm run test:services && sleep 3 && npm run test:e2e"
  }
}

What I Learned

Testing Distributed Systems is Different

Unit tests give you confidence in individual functions, but they don’t tell you if your system works as a whole. E2E tests with real infrastructure (database, message queue) are essential for distributed systems.

Don’t Mock Infrastructure

Initially, I thought about mocking PostgreSQL and SQS. But the bugs often happen in the interactions with real infrastructure. Connection pooling issues, transaction isolation levels, message queue delivery semantics - these only surface with real systems.

Using Docker Compose and LocalStack lets you run real infrastructure locally without AWS costs.

The waitFor Pattern is Essential

Distributed systems are eventually consistent. Your tests need to poll and wait for conditions to become true. Don’t use fixed sleep() calls - they make tests slow and flaky.

Test Failure Scenarios Explicitly

The most valuable tests aren’t the happy path - they’re the failure scenarios:

What if the payment service crashes?
What if the database transaction rolls back?
What if events are delivered twice?

These tests caught real bugs in my implementation that unit tests missed.

The Value of Real Implementation

Going through this exercise made me understand the transactional outbox pattern at a much deeper level than just reading about it. I encountered real issues:

Race conditions: Two payment service instances trying to process the same event
Connection pooling: Database connections exhausting under load
Message ordering: Events processing out of order
Cleanup: Outbox table growing unbounded

Each of these forced me to think about the real-world implications of the pattern.

The Complete Implementation

I’ve built a complete working example of this system that you can clone and run locally. The repository includes:

Full TypeScript implementations of both services
Complete test suite with E2E, idempotency, and failure scenario tests
Docker Compose setup to run everything locally
Detailed README with setup instructions

Local Results

GitHub Repository: github.com/CodeJonesW/flight-booking-system

Key files to check out:

services/flight-service/src/ - Flight service implementation
services/payment-service/src/ - Payment service implementation
tests/e2e/ - All E2E tests
docker-compose.yml - Infrastructure setup

You can have this running on your machine in about 5 minutes with:

git clone https://github.com/CodeJonesW/flight-booking-system
cd flight-booking-system
docker-compose up -d
npm install
cd services/flight-service && npm install && cd ../..
cd services/payment-service && npm install && cd ../..
npm test

Resources

Tools Used:

Docker Compose - Orchestrate services locally
LocalStack - Mock AWS services locally
Jest - Testing framework
node-postgres - PostgreSQL client
AWS SDK for JavaScript - SQS client

Further Reading:

Testcontainers - Alternative to docker-compose for tests
Pattern: Transactional Outbox - More details on the pattern
Testing Microservices, the sane way - Great article on testing strategies

Cheers!

Jones Codes

Explorer

Testing Distributed Systems: Beyond Unit Tests

The Testing Challenge

The Local Testing Stack

Project Structure

Setting Up Local Infrastructure

Docker Compose Configuration

Database Schema

LocalStack Setup

Core Implementation Concepts

Flight Service - Transactional Outbox

Event Processor - Polling and Publishing

Payment Service - Idempotent Processing

Writing E2E Tests

The waitFor Pattern

Test Structure

Testing Idempotency

Testing Failure Scenarios

Running the Tests

What I Learned

Testing Distributed Systems is Different

Don’t Mock Infrastructure

The waitFor Pattern is Essential

Test Failure Scenarios Explicitly

The Value of Real Implementation

The Complete Implementation

Local Results

Resources

Backlinks

Graph View

Recent Posts

From LRU Cache to Distributed Systems: A Complete Guide to Caching in Modern Applications

Posts

Testing Distributed Systems: Beyond Unit Tests

Solving the Dual Write Problem - Transactional Outbox and Idempotency

Cache Invalidation in Microservices: The Hard Part