Store snapshots in S3

A recorder stores snapshots of a Machine's state. MemoryRecorder, the one that ships, keeps them in the process, which means they are gone on restart and the Machine replays its whole history to catch up. Persisting snapshots somewhere outside the process fixes that, and lets you step back through past states across runs rather than only within one.

This guide writes them to an S3 bucket. The same shape works for any store: the Machine only cares about four methods.

What a recorder has to implement

Four methods. The signatures below are the ones @mintblue/sdk 9.6.0 declares, with the package's own doc comments left in the declaration file rather than reproduced here:

export interface Recorder {
    afterDispatch(snapshot: Snapshot): Promise<void>;
    getSnapshots?(): Promise<Snapshot[]>;
    createSnapshot?(snapshot: Snapshot): Promise<void>;
    loadSnapshot?(id?: string): Promise<Snapshot | null>;
}

The optional three are opt-in by presence. Omit loadSnapshot and there is no persistence to resume from; omit createSnapshot and explicit snapshots are disabled.

A snapshot is the state plus the action that produced it. The package declares it verbatim as:

export interface Snapshot {
    /** The resulting state of the application after the action was dispatched. */
    state: AppState;
    /**
     * The action that was dispatched, leading to the current state.
     * If action is undefined, it means the state in the Snapshot is the initial state
     * because no action was executed to reach this state
     */
    action: Action | undefined;
    /**
     * The context in which the action was dispatched. This can contain
     * additional information or metadata about the current environment or situation.
     */
    ctx?: Context;
}

One rule the declaration states and every recorder has to respect: a recorder must not change the snapshot it receives. Clone it first if you need to.

Before you start

A bucket, and credentials with permission to read and write objects in it. This guide uses the AWS SDK v3 client.

npm install @aws-sdk/client-s3

Credentials come from the environment. Do not put them in the file.

Step 1: Two helpers for the bucket

const {
  GetObjectCommand,
  PutObjectCommand,
  S3Client,
} = require('@aws-sdk/client-s3');

const s3 = new S3Client({
  region: process.env.AWS_REGION,
  endpoint: process.env.AWS_ENDPOINT,
});

async function upload(bucket, key, body) {
  await s3.send(new PutObjectCommand({ Bucket: bucket, Key: key, Body: body }));
}

async function download(bucket, key) {
  return s3.send(new GetObjectCommand({ Bucket: bucket, Key: key }));
}

module.exports = { upload, download };

endpoint is only needed when the bucket is behind an access point of your own. Leave it unset for AWS itself.

Step 2: Write the recorder

Four methods, and one decision inside the first one.

const { upload, download } = require('./s3.js');

class S3Recorder {
  constructor(file, bucket) {
    this.file = file;
    this.bucket = bucket;
    this.snapshot = { action: undefined, state: {} };
    this.timer = undefined;
  }

  async afterDispatch(snapshot) {
    if (!snapshot.action) {
      return;
    }

    this.snapshot = snapshot;

    if (this.timer) clearTimeout(this.timer);
    this.timer = setTimeout(() => this.createSnapshot(snapshot), 5000);
  }

  async getSnapshots() {
    return [this.snapshot];
  }

  async createSnapshot(snapshot) {
    const body = JSON.stringify(snapshot, null, 2);
    await upload(this.bucket, this.file, new TextEncoder().encode(body));
  }

  async loadSnapshot() {
    try {
      const response = await download(this.bucket, this.file);
      if (!response.Body) return null;
      return JSON.parse(await response.Body.transformToString());
    } catch (error) {
      if (error.name === 'NoSuchKey') return null;
      throw error;
    }
  }
}

module.exports = { S3Recorder };

Three things in that class are deliberate.

The first snapshot is skipped. A snapshot with no action is the initial state, before anything was dispatched. There is nothing to persist.

A missing object is not an error. The first run has nothing in the bucket yet, and loadSnapshot returning null is how the interface says so. Letting the NoSuchKey error out of the method would stop a Machine that has simply never run before.

The upload is debounced. afterDispatch runs after every dispatch, and a Machine catching up on history dispatches as fast as it can read. Uploading on each one turns a resynchronisation into thousands of writes. Holding the latest snapshot and uploading five seconds after the last dispatch means one write per quiet moment.

A recorder that debounces is a recorder that can be behind. If your process can stop between the last dispatch and the timer firing, either shorten the delay or call machine.createSnapshot() at the points that matter to you.

Step 3: Use it

The recorder is the fourth constructor argument, in place of MemoryRecorder.

const {
  Machine,
  MintblueReader,
  MintblueWriter,
} = require('@mintblue/sdk');
const { Consignments } = require('./consignments.js');
const { S3Recorder } = require('./s3-recorder.js');

async function main() {
  const sdkToken = process.env.MINTBLUE_SDK_TOKEN;
  const projectId = process.env.MINTBLUE_PROJECT_ID;

  const machine = new Machine(
    Consignments,
    await MintblueReader.create(sdkToken, projectId),
    await MintblueWriter.create(sdkToken, projectId),
    new S3Recorder('consignments.json', process.env.SNAPSHOT_BUCKET),
    { emitAppEvents: true },
  );

  await machine.loadSnapshot();
  machine.start();
}

main();

machine.loadSnapshot() calls your loadSnapshot and loads the returned state into the application instance before the reader starts, which is what turns a persisted snapshot into a faster start.

Check it worked

Three checks, and the third is the one that proves persistence rather than writing.

1. An object appears in the bucket. After a dispatch and five seconds of quiet, consignments.json exists and its JSON has a state and an action.
2. createSnapshot can be triggered on demand. Call await machine.createSnapshot() and the object is rewritten immediately.
3. A restart resumes from it. Stop the process, start it again, and log machine.getInstance().state straight after loadSnapshot() and before start(). The state is the persisted one, not empty.

await machine.loadSnapshot();
console.log(machine.getInstance().state);
machine.start();

If that logs the batches recorded in the previous run, the snapshot did its job: the Machine started from stored state instead of rebuilding it from the whole history.

Next