Your agent writes the code. You review ten sentences.
However fast an LLM writes code, whether that code still does what the spec says
is a separate question. Belay, a black-box scenario harness for vibe coding, puts the system
under test in src/ untouched, and builds
executable specs in test/ that only ever knock from the outside, through interfaces a
user can reach. A browser screen, a gRPC stream — every surface a person touches counts.
A spec that existed only as prose becomes a contract that actually runs, every time.
And because that contract is written as sentences rather than code, the artifact
you have to read line by line is the spec, not the diff.
.feature
TypeScript step definitions
HTTP · GraphQL · gRPC · WebSocket
Browser control · capture · visual checks
Mock infra · fault injection · clock control
Language and framework agnostic
01 Getting started
Install once, run belay init once. After that it is just belay run.
Belay installs as a global CLI. The harness and the system under test never share a runtime,
so it does not matter what src/ is written in. Node is what Belay runs on — it is
not a constraint on your project.
Requirements
| Item | Needed | Notes |
|---|---|---|
| Node.js 22+ | required | For the Belay CLI and step definitions. Unrelated to src/, which can be any language |
| Docker | optional | For containerised infra. Without it, stub and process mode still give you a full environment |
| Browser engine | optional | Only if you pick the browser surface; init downloads just what you selected |
Install
terminalnpm install -g @beoks/belay
belay --version
belay doctor # check ports, Docker and engines up front
Initialise
The command is the same whether you start from an empty directory or a project that is already
running. init looks around and proposes what it finds as the defaults.
mkdir my-project && cd my-project
belay init
# Press enter to accept the value in brackets.
# detected: npm --prefix src start
Surfaces to verify (comma separated: http, graphql, ws, browser, cli)
[http] > http, ws
Command that starts src/
[npm --prefix src start] >
URL that proves it is ready
[http://localhost:8080/health] >
Add an external API stub? (in-process, no Docker)
[y/N] > y
Add a mail capture server? (SMTP)
[y/N] > y
✓ src/ # created if absent — where the system under test lives
✓ belay.config.ts # prefilled with the surfaces and infra you chose
✓ test/support/world.ts
✓ test/support/steps/common.steps.ts
✓ test/features/L0-smoke/health.feature # the first anchor
✓ test/features/L0-smoke/health.steps.ts
✓ AGENTS.md # the rule that keeps agents out of test/
✓ .github/workflows/belay.yml
✓ .gitignore
✓ package.json # test · test:smoke · test:watch scripts
Next steps
1. Put your project in src/ and check app.start in belay.config.ts
2. belay run --only L0 # get the first anchor green
3. belay new L1-<domain>/<feature> # pin a spec as a scenario
What gets created
my-project/
├── belay.config.ts # prefilled from the surfaces and infra you chose
├── AGENTS.md # rules for agents, including "test/ is read-only"
├── src/ # your existing code moves here wholesale
└── test/
├── support/ # world · hooks · drivers for the surfaces you picked
├── contracts/ # schema files, if you chose GraphQL or gRPC
├── baselines/ # created only if you chose the browser
└── features/
└── L0-smoke/
└── health.feature
The first scenario it writes is deliberately trivial.
test/features/L0-smoke/health.featureFeature: service boots
The first anchor: proof that the harness itself is alive.
Once green, the whole path works — infra up, build, readiness probe,
scenario execution, and teardown.
Scenario: it answers the health check
When I GET "/health"
Then the response succeeds
src/ built, the readiness probe passed, a scenario executed,
and everything came back down. Every other scenario is stacked on top of that — it is checking
the anchor is in before you weight the rope.
Commands
| Command | What it does |
|---|---|
belay init | Scaffold the config and the test/ skeleton. Safe to run over an existing project |
belay run | The whole thing — infra up, scenarios, teardown |
belay run --upto L1 | Cut the run at a layer for fast feedback (--only L2 also works) |
belay run --grep signup | Filter scenarios by name (--tags "@smoke" too) |
belay run --watch | Re-run on change. The environment stays up, so a scenario edit re-executes in milliseconds |
belay run --bail | Stop at the first failure |
belay up | Boot the environment and hold it in the foreground, for poking by hand |
belay down | Clean up an environment kept after a failure (child processes, containers) |
belay new L2-order/checkout | Create a .feature and its step file together |
belay approve cart-empty | Accept a visual baseline, after showing you the diff |
belay report --open | Open the latest report |
belay doctor | Check ports, Docker, browser engines and orphaned containers |
Language
Messages, CLI help and the generated scaffolding all follow BELAY_LANG
(en, ko), falling back to your locale and then to English.
belay init # English Gherkin, steps and AGENTS.md
BELAY_LANG=ko belay init # the same, in Korean
The Gherkin parser reads en, ko and ja dialects regardless of
that setting. A suite written in Korean runs fine for someone whose tooling speaks English, and the
repository's own examples use both, so each dialect is exercised on every CI run.
Handing it to an agent
init writes an AGENTS.md whose rules are short. The point of all of them is
one thing — keep the examiner and the candidate apart.
## Belay harness rules
- Change the implementation only inside `src/`.
- `test/` is read-only. Never edit a scenario to make it pass.
- Run `belay run` before reporting that you are done, and attach the output verbatim.
- If you believe a spec is wrong, do not change it — report it with your reasoning.
With that in place the loop is three lines. The agent edits src/, belay run
decides, and the failure list is the todo list. It cannot rewrite the grading criteria, so the only
thing that can converge is the implementation. That is what safe vibe coding means here.
belay run --only L0 # check the anchor
belay new L1-account/signup # pin the spec as a scenario first
belay run # the failure list is the remaining work
02 Adopting a project you already have
The projects that most need a safety net are the ones already in flight.
Belay does not need a fresh start. Point it at the service you are already running, and it reads the code to fill in the parts you would otherwise have to describe from scratch.
terminalbelay init --from ../my-api --link symlink
✓ src → ../my-api
Read from the project
stack node
start npm start --prefix src (high: scripts.start = "node src/index.js")
port 4321 (high: PORT=4321)
health /healthz
surfaces http
infra db · cache (from compose — declare what you need)
The port, the start command and the health path all come out of the code, so the first scenario knocks on the endpoint this service actually serves rather than a convention it may not follow. Every reading carries the evidence it came from — a proposal you can check beats one you have to trust.
How src/ connects
The four modes differ in who owns the code afterwards, and that decides whether CI can even check the suite out. That is not a choice to make on someone's behalf.
--link | What happens | Use when |
|---|---|---|
symlink | The project stays where it is; edits apply immediately | You are still working on it locally |
submodule | Pins a commit, so a run states which revision it passed against | It is a shared repository, or CI must check it out |
move | The harness becomes the project repository | You want one repository |
copy | A snapshot, which will drift from the original | You want a frozen reference |
A symlinked src is added to .gitignore: it points outside the
repository, and committing that helps nobody. A submodule is committed, and CI needs
actions/checkout@v4 with submodules: true.
Turning existing code into scenarios
belay analyze reads the connected project and writes a checklist to
.belay/analysis.md: the endpoints it found, grouped into the domains that become
L1- directories, with a success candidate and a refusal candidate for each.
### `L1-orders`
| Surface | Operation | Where |
|---|---|---|
| http | `GET /api/orders` | src/routes/orders.js |
| http | `POST /api/orders` | src/routes/orders.js |
| http | `ANY /api/orders/:param` | src/routes/orders.js |
Candidates:
- [ ] `POST /api/orders` — the success path: what is true afterwards that was not before?
- [ ] `POST /api/orders` — the refusal: which input is rejected, and with which error code?
POST /api/orders; it cannot tell you that placing
an order must not charge the customer twice. That sentence has to come from you, and a tool
that pretended otherwise would be handing you a suite that looks complete and checks nothing.
Skills for the agent doing the work
init writes the same skill to .claude/skills/belay/ and
.agent/skills/belay/, so a coding agent loads the boundary before it starts:
it may rewrite src/, and it may not edit test/ to make something
pass. Belay's whole value rests on that separation, and an agent will not honour it from a
README it was never asked to read.
belay run --only L0 before writing any
scenario. If the harness cannot boot your project yet, every later failure will be about
that and not about your code.
03 Why Belay
The bottleneck in vibe coding is not how fast you produce code. It is the fear of regression — and the cost of reading what the agent wrote.
Hand a feature to an agent and the code appears. The problem starts immediately after. Is it safe to let it refactor? To switch frameworks? Did this commit quietly break the checkout flow written two months ago? With no way to answer, someone ends up clicking through it by hand every time — and the speed you gained disappears there.
Each of the usual answers leaks, for a different reason.
Specs that live only as documents
They never execute. When the implementation betrays them nothing says so, and given time the document rots first.
Unit tests bound to internals
Coupled to function signatures and class names. Change the structure and the tests break first — the very thing meant to enable refactoring blocks it.
Manual QA
Not reproducible, not recorded. If it cannot run on every commit it is not a safety net.
Executable specs, kept outside the implementation
Scenarios live outside src/. Replace everything inside and they remain, as the answer key.
Reading the code is the other half of the bottleneck
Vibe coding did not only make code cheaper to write; it made it more expensive to trust. An agent hands you a five-hundred-line diff in a minute, and approving it by inspection means reading all five hundred lines — again on the next commit, and the one after that. The artifact you have to review grows exactly as fast as the thing that made you fast.
Belay hands you a smaller artifact instead, written in a language you already read.
test/features/L2-cart/coupon.feature Scenario: a coupon that has been used cannot be used again
Given the coupon "WELCOME" has already been redeemed
When I apply it to my cart
Then the request is refused
And the cart total is unchanged
That is what goes under review, line by line. No framework, no class names, no control flow — only the behaviour you are agreeing to. Ten sentences pin down what a thousand lines of implementation are allowed to do, and they stay legible to the person who asked for the feature in the first place.
And these sentences are not a description of the code, they are a check on it. The steps knock only on surfaces a user can reach, so a green run means the sentences are true of the running system — not that something was mocked into agreeing with them. Review the spec; let the suite review the implementation.
04 The idea
The climber moves freely. The belayer holds the rope.
Climber = src/
Where the agent climbs. Language, framework and architecture are all free to change.
Belayer = test/
Holds the rope from outside. It does not know the internals and does not need to. It only catches the fall.
Anchor = the run contract
What to boot and how to knock on it, pinned in one file. Without it the rope holds nothing.
Three rules follow, and they are the whole of Belay.
- Black box. Tests reach the system only through interfaces the deployed artefact exposes. No importing source, no calling internals, no touching its database.
- Reproducible. Everything from booting infrastructure to cleaning up lives in one command. What works on your laptop has to be what works in CI.
- The spec is the test. A
.featurefile is both the specification a person reads and the check a machine runs. There is no gap for them to drift into.
05 Project layout
The top level is always just src and test.
# project root
Belay/
├── belay.config.ts # the anchor: how to boot src/ and how to knock on it
├── docs/
│ └── index.html
│
├── src/ # ── the system under test, exactly as its own repo would look ──
│ ├── package.json # Node? Kotlin? Go? Rust? It makes no difference
│ ├── Dockerfile
│ └── ... # Belay never looks inside
│
└── test/ # ── black-box scenarios ──
├── support/
│ ├── world.ts # per-scenario context (drivers, state)
│ ├── hooks.ts # Before / After, isolation between scenarios
│ ├── steps/ # shared steps (status codes, JSONPath assertions)
│ └── drivers/ # surface adapters — no domain logic
│ ├── http.ts
│ ├── graphql.ts
│ ├── grpc.ts
│ ├── ws.ts # WebSocket / SSE and other streaming
│ ├── browser.ts # page control · capture · visual checks
│ ├── cli.ts
│ └── mocks.ts # programming stubs · asserting on what they received
│
├── contracts/ # protocol schemas — where drivers get their types
│ ├── schema.graphql
│ ├── order.proto
│ └── pg.openapi.yaml # the external payment stub's contract
│
├── baselines/ # approved screenshots — changes go through review
│ └── L2-checkout/
│ ├── cart-empty.desktop.png
│ └── cart-empty.mobile.png
│
└── features/
├── L0-smoke/
│ └── health.feature
├── L1-account/
│ ├── signup.feature # HTTP
│ ├── signup.steps.ts
│ └── login.feature
├── L1-catalog/
│ ├── search.feature # GraphQL
│ └── search.steps.ts
├── L2-order/
│ ├── checkout.feature # gRPC + WebSocket
│ └── checkout.steps.ts
├── L2-ui-checkout/
│ ├── cart.feature # browser journey + visual checks
│ └── cart.steps.ts
└── L3-cross/
└── signup-to-first-order.feature
| Path | Role | Rule |
|---|---|---|
src/ | The system under test | No constraints on its contents. Replaceable wholesale |
test/features/ | Scenario layers | Directories are layers. Keep a .feature and its steps together |
test/support/ | Drivers and context | Surface adapters only. No domain logic |
test/contracts/ | Protocol schemas | GraphQL SDL, .proto, OpenAPI. Where driver types come from |
test/baselines/ | Approved screenshots | Committed. Updated only through explicit, reviewed approval |
belay.config.ts | The run contract | Infra, boot, readiness, surfaces, reports, cleanup |
06 The execution contract
One file at the root holds everything needed to put this project into a testable state.
This file is the only thing Belay knows about src/. How to build it, what it depends on,
when to consider it ready, and which windows to knock on. Anything not written here is invisible to
the tests — which is the point.
import { defineConfig } from '@beoks/belay';
export default defineConfig({
// 1. Whatever src/ needs around it — containers, mock servers, anything
infra: [
{ name: 'postgres', image: 'postgres:16', ports: ['5432'],
env: { POSTGRES_PASSWORD: 'test' },
ready: { tcp: '5432' } },
{ name: 'payment-gateway', // stand-in for the payment provider
stub: { port: 4010 },
// A programmable in-process stub: starts in milliseconds, needs no Docker },
],
// 2. How to boot src/. One command is enough
app: {
build: 'docker build -t belay-sut ./src',
start: 'docker run --rm -p 8080:8080 --env-file .belay/env belay-sut',
env: {
DATABASE_URL: 'postgres://postgres:test@localhost:5432/app',
PAYMENT_BASE_URL: 'http://127.0.0.1:4010',
},
// Without a readiness probe every run races the boot
ready: { http: 'http://localhost:8080/health', timeoutMs: 60_000 },
},
// 3. The only windows tests may knock on.
// What is declared here exists; reaching the SUT any other way breaks the black box.
surfaces: {
http: { baseUrl: 'http://localhost:8080' },
graphql: { endpoint: 'http://localhost:8080/graphql' },
grpc: { address: 'localhost:9090', proto: './test/contracts/order.proto' },
ws: { url: 'ws://localhost:8080/ws' },
cli: { command: 'docker run --rm belay-sut app-cli' },
// The front-end surface — the screen a person actually sees
browser: { baseUrl: 'http://localhost:3000',
engine: 'chromium',
viewports: { desktop: [1280, 800], mobile: [390, 844] },
locale: 'en-US', timezone: 'UTC', colorScheme: 'light' },
},
// 4. Visual policy
visual: {
baselineDir: 'test/baselines',
diff: { threshold: 0.01 },
interpret: { enabled: true, gate: false }, // never gates by default — see 10
capture: { onFailure: ['screenshot', 'dom', 'console'] },
},
// 5. Control — declared windows onto state you cannot create from outside
control: {
clock: { via: 'http', endpoint: '/__test/clock' }, // a test-only contract the SUT serves
seed: { via: 'http', endpoint: '/__test/seed' },
faults: { targets: ['postgres', 'payment-gateway'] }, // what may be slowed or cut
},
// 6. Where the scenarios are
suite: {
features: 'test/features/**/*.feature',
steps: 'test/**/*.steps.ts',
support: 'test/support/**/*.ts',
},
// 7. Isolation between scenarios — leaked state eats trust
isolation: {
between: 'scenario',
// Either a shell command or a request to a declared window. Neither touches internals.
reset: { http: '/__test/seed', body: { data: { todos: [] } } },
browser: true, // a fresh context every time — cookies, storage, session
mocks: true, // stub programming and recorded calls both cleared
},
// 8. Deadlines — so one stuck step cannot hold the run hostage
timeouts: { step: 30_000, scenario: 120_000, hook: 30_000 },
// 9. Results and cleanup
report: { formats: ['html', 'junit', 'json'], outDir: '.belay/reports', trace: true },
teardown: { keepOnFailure: true },
});
belay run — typed by a person, by CI or by an
agent, the same command, the same order, the same result. If part of the setup lives in someone's head,
that suite is not something to trust.
07 Run lifecycle
What one belay run does — go up, stay protected, come back down.
belay.config.ts and check port conflicts, required tools and leftovers from a previous run. If this fails, nothing is started.ready condition holds.app.build → app.start. The environment it receives points at the infrastructure just started. The SUT has no idea it is under test.app.ready must pass before anything proceeds. On timeout the boot log is attached to the failure — telling "it was slow" apart from "it crashed" is stated, not guessed..belay/reports.keepOnFailure on, a failed run's environment is left standing for a post-mortem.08 Writing scenarios
Written in Gherkin, so the specification a person reads and the check a machine runs are the same sentences — which is what makes the spec reviewable by someone who will never open src/.
Feature: email signup
An unused email creates an account and sends a verification mail.
An email already in use is refused.
Background:
Given the mail stub is empty
Scenario: signing up with a new email succeeds
Given "ada@example.com" is not registered
When I sign up as "ada@example.com"
Then the status is 201
And the response contains an account id
And 1 verification mail is sent to "ada@example.com"
Scenario: an email already in use is refused
Given an account exists for "ada@example.com"
When I sign up as "ada@example.com"
Then the status is 409
And no verification mail is sent
Scenario Outline: malformed emails are refused
When I sign up as "<input>"
Then the status is 400
Examples:
| input |
| not-an-email |
| @example.com |
| |
What makes a scenario good
Say only what is observable
"the status is 409", "1 verification mail is sent" — facts checkable from outside.
Describe the implementation
"UserService.create is called", "a row appears in the users table" — statements that break when the structure changes.
Make each scenario self-contained
Never lean on leftovers from the previous one. Shuffle the order and the result must not change.
Write the failure paths too
A spec with only happy paths is the most dangerous kind when porting. Pin refusals, duplicates and timeouts as well.
09 Step definitions
Always TypeScript. Whatever language src/ is, the belayer speaks one.
Step implementations should be thin. Translate the sentence into a driver call, assert on the response, and stop there. Reimplementing domain rules here turns the test suite into a second implementation of the system.
test/features/L1-account/signup.steps.tsimport { Given, When, Then, expect } from '@beoks/belay';
import type { AppWorld } from '../../support/world';
Given('{string} is not registered', async function (this: AppWorld, email: string) {
// Check through the public API only. Never go digging in the database.
const res = await this.http.get(`/api/v1/accounts?email=${encodeURIComponent(email)}`);
expect(res.status).toBe(404);
this.ctx.email = email;
});
Given('an account exists for {string}', async function (this: AppWorld, email: string) {
const res = await this.http.post('/api/v1/accounts', { email, password: 'P@ssw0rd!' });
expect(res.status).toBe(201);
this.ctx.email = email;
await this.mock.mail.clear(); // side effects of setup are not under observation
});
When('I sign up as {string}', async function (this: AppWorld, email: string) {
this.last = await this.http.post('/api/v1/accounts', { email, password: 'P@ssw0rd!' });
});
Then('the status is {int}', function (this: AppWorld, status: number) {
expect(this.last.status).toBe(status);
});
Then('{int} verification mail is sent to {string}',
async function (this: AppWorld, count: number, to: string) {
// The mail stub is ours, so reading its inbox keeps the black box intact.
const mails = await this.mock.mail.waitFor({ to, timeoutMs: 5_000 });
expect(mails).toHaveLength(count);
});
test/support/world.ts
import { World, setWorldConstructor } from '@beoks/belay';
export class AppWorld extends World {
// World already carries a driver per surface:
// this.http · this.gql · this.grpc · this.ws · this.cli · this.page
// this.mock.<name> — the stubs and mail servers you declared
// this.control — clock · seed · faults
// this.last — the most recent Exchange, which shared steps assert on
// Touching a surface that is not configured fails, telling you what to declare.
// Keep only scenario-scoped state and thin helpers here.
token?: string;
async authenticate(email: string, password = 'P@ssw0rd!'): Promise<void> {
const response = await this.http.post('/api/v1/sessions', { email, password });
this.token = (response.body as { token?: string })?.token;
}
}
setWorldConstructor(AppWorld);
surfaces.
The moment step code imports from src/ or execs into a container, it stops being a black box,
and passing no longer says anything about a port to another stack.
10 Protocol drivers
HTTP · GraphQL · gRPC · WebSocket — different surfaces, the same shape of sentence.
Backends rarely speak one protocol. Reads over GraphQL, internal calls over gRPC, live updates over
WebSocket, admin over REST. If each protocol brought its own way of asserting, the scenarios would turn
into technical documentation and collapse along with the next rewrite of src/.
So every Belay driver returns the same Exchange. Status, body and metadata
are normalised into one shape, and assertions about status codes or JSON paths are written once and
reused across protocols.
| Protocol | Driver call | What the scenario observes |
|---|---|---|
| HTTP | this.http.post(path, body) | Status, headers, body, elapsed time |
| GraphQL | this.gql.query(doc, vars) | data shape, errors[].extensions.code, partial success |
| gRPC | this.grpc.call('svc/Method', msg) | Status code, trailers, the response message |
| gRPC stream | this.grpc.stream(...) | Message order, terminal status, what must not arrive |
| WebSocket | this.ws.connect() → waitFor() | Messages received, order, reconnection, close code |
| Queue | this.queue.publish(topic, msg) · consume(topic) → waitFor() | Messages published, keys and headers, order, what must not arrive (quietFor) |
| CLI | this.cli.run(args) | Exit code, stdout, stderr |
Exchange normalisation, are implemented and exercised by the examples on every run
(gRPC in examples/04-grpc — unary, status codes, server streaming). The gRPC surface
needs optional peers, @grpc/grpc-js and @grpc/proto-loader; the queue
surface needs kafkajs. The queue driver creates missing topics through the admin API,
so it does not depend on broker auto-create. Its verification status is stated honestly in
section 17 — field-verified against a live broker, not continuously verified by an example.
The normalised exchange
test/support/drivers/types.tsexport interface Exchange<T = unknown> {
ok: boolean;
status: number; // HTTP status · gRPC code · WS close code, normalised
body: T; // data for GraphQL, the message for gRPC
errors?: { code: string; message: string; path?: string[] }[];
meta: Record<string, string>; // headers · trailers · frame metadata
elapsedMs: number;
// Every Exchange is traced automatically — no logging to add
}
Which means these steps are written once and used everywhere.
test/support/steps/common.steps.tsThen('the response succeeds', function (this: AppWorld) {
expect(this.last.ok).toBe(true);
});
Then('the error code is {string}', function (this: AppWorld, code: string) {
expect(this.last.errors?.[0].code).toBe(code); // GraphQL · gRPC · HTTP alike
});
Then('the response {string} is {string}',
function (this: AppWorld, path: string, value: string) {
expect(String(readPath(this.last.body, path))).toBe(value);
});
gRPC keeps both halves of the contract
A refusal has two answers: what the transport said, and what the domain meant. Porting has to preserve
why something was refused, not just that it was — so Belay keeps the transport verdict in
meta and lifts a domain code out of the details string into errors[0].code.
Scenario: reserving more than available is refused
When I reserve 9 of "CAM-001"
Then the gRPC status is "FAILED_PRECONDITION"
And the error code is "OUT_OF_STOCK"
Streaming has a time axis
test/features/L2-order/checkout.steps.tsGiven('I am subscribed to the order channel', async function (this: AppWorld) {
this.sub = await this.ws.subscribe({ topic: 'order', id: this.ctx.cartId });
});
Then('I receive order status {string} within {int} seconds',
async function (this: AppWorld, status: string, sec: number) {
// A condition with a deadline, not a sleep: fixed waits are how suites go flaky.
const msg = await this.sub.waitFor(
(m) => m.type === 'order.status' && m.status === status,
{ timeoutMs: sec * 1000 },
);
expect(msg.orderId).toBe(this.last.body.orderId);
});
Then('no other order notification arrives', async function (this: AppWorld) {
await this.sub.quietFor({ ms: 1_000 });
});
waitFor also scans what already arrived, so a message landing between the action and the
assertion is not missed. And "nothing should arrive" only means something once you have actually waited
for it — that is what quietFor is.
11 Browser and visual checks
A front end's public interface is the screen. So Belay opens it, clicks it, captures it and reads it.
For a front-end project, "black box" means the browser. Belay drives a real engine the way a person would and checks the result at three levels. Each level has a different degree of determinism — and therefore a different right to fail your build.
1 · Structural assertions
Roles, names and states read from the accessibility tree. Fully deterministic, so this is always the first basis for pass or fail.
2 · Visual regression
Perceptual diff against an approved baseline. Thresholds are declared, so it is reproducible — on the machine that approved it.
3 · Visual interpretation
A model reads the capture and judges claims like "the legend does not overlap". Catches what no baseline covers, but is not deterministic.
examples/05-browser — semantic locators, capture and baseline comparison. The visual gate was
checked by breaking it on purpose: changing one CSS colour produced a 3.33% difference and failed the run.
It needs optional peers: npm i -D playwright pngjs pixelmatch && npx playwright install chromium
visual.interpret.gate — the model's reasoning is recorded alongside the verdict.
The scenario
test/features/L1-cart/cart.featureFeature: the cart page
Scenario: an empty cart explains itself and refuses checkout
Given the cart is empty
When I open the cart page
Then I see "Your cart is empty"
And the "Checkout" button is disabled
Scenario: changing the quantity updates the total
Given the cart holds 1 of "CAM-001"
When I open the cart page
And I set the quantity to 3
Then the total reads "$387.00"
And no console errors occurred
@visual
Scenario Outline: the checkout button stays reachable on every viewport
Given the cart holds 2 of "CAM-001"
When I resize the window to "<viewport>"
And I open the cart page
Then the "Checkout" button is enabled
And the screen "cart-filled-<viewport>" matches its baseline
Examples:
| viewport |
| desktop |
| mobile |
Browser steps
test/features/L1-cart/cart.steps.tsWhen('I set the quantity to {int}', async function (this: AppWorld, quantity: number) {
// Found by label, never by CSS class: a class disappears with the framework.
await this.page.byLabel('Quantity').first().select(String(quantity));
await this.page.waitForStable();
});
Then('the {string} button is disabled', async function (this: AppWorld, name: string) {
await this.page.byRole('button', { name }).expectDisabled();
});
// ── deterministic: this one gates ─────────────────────────────
Then('the screen {string} matches its baseline',
async function (this: AppWorld, name: string) {
await this.page.waitForStable(); // animations, fonts and images settled
const shot = await this.page.capture(name);
await shot.toMatchBaseline(); // missing baseline fails; approval is a separate act
});
Then('no console errors occurred', function (this: AppWorld) {
expect(this.page.consoleErrors()).toEqual([]);
});
Locator rules
On the front end, keeping the black box intact comes down to what you find elements by. Only criteria that survive a move from React to Svelte let the scenarios go on being the answer key.
| How you locate | Verdict | Why |
|---|---|---|
byRole('button', { name }) | preferred | Matches how a user perceives it, and forces accessibility to exist |
byLabel · byPlaceholder | preferred | The semantic identifiers of form controls |
byTestId('cart-total') | fallback | Only where a role cannot express it. It is a contract, so a port must keep it |
.css .class > div:nth-child(2) | never | Breaks on a style change; wiped out entirely by a framework swap |
| Component instances or internal stores | never | A black-box violation, and a concept that does not exist outside the browser |
Front-end-only projects
When src/ is only a front end, contract-based mocks stand in for the backend.
Start stubs from the OpenAPI or GraphQL schema in test/contracts/, point the browser's network
at them, and every loading, error, empty and slow state becomes reproducible without a server.
// Shaping the backend's answer to reach a screen state — the stub is ours, so this is allowed
Given('the product list is delayed by {int}ms', async function (this: AppWorld, ms: number) {
await this.mock.api.onGet('/api/v1/products').delay(ms);
});
Given('the product list fails', async function (this: AppWorld) {
await this.mock.api.onGet('/api/v1/products').reply(503);
});
// → lets you pin a screen contract like "then a retry button is shown", deterministically
12 Controlling the environment
Producing situations you cannot reach from outside — without reaching inside.
The things most worth checking usually cannot be reproduced with an ordinary request. A coupon that expires in thirty days. The moment the payment provider returns 502. What retries do when the database blinks. The common temptation here is "let the test write to the database directly" — and portability dies right there.
Belay uses control surfaces instead. There are two kinds of target, and they have different rules.
Mock infra — ours
External API stubs, brokers and test databases were started by Belay. Change their answers, kill them, and assert on what they received.
The SUT — not ours
Moving its clock or seeding it goes through a test-only contract declared under control. No exec into the container, no writing to its state.
| Control | Call | What it lets you verify |
|---|---|---|
| Clock | this.control.clock.advance('P30D') | Expiry, schedules, grace periods, settlement cutoffs |
| Seed | this.control.seed('catalog.yaml') | Large-data preconditions, pagination |
| Stub response | this.mock.pg.reply(502) | Fallback, retry and user messaging under external failure |
| Stub latency | this.mock.pg.delay(5_000) | Timeout handling, loading states, circuit breakers |
| Received calls | this.mock.pg.received() | Idempotency, retry counts, the payload actually sent |
| Faults | this.control.faults.partition('pg') | Reconnection, health checks, data consistency |
| Network | this.control.faults.latency('pg', 800) | Behaviour and timeout boundaries under a slow dependency |
Feature: surviving a payment outage
Scenario: a transient outage holds the order and charges exactly once
Given items are in the cart
And the payment provider answers 502 to the first 2 requests
When I place the order
Then the response succeeds
And I receive order status "PENDING_PAYMENT" within 30 seconds
And every charge that reached the provider carries the same idempotency key
Scenario: an unpaid order is cancelled after three days
Given an order in "PENDING_PAYMENT"
When 3 days pass
Then the order status is "CANCELLED"
And the stock is restored
Given('the payment provider answers {int} to the first {int} requests',
async function (this: AppWorld, status: number, times: number) {
await this.mock.pg.onPost('/charges').replyTimes(times, status).reply(200);
});
When('{int} days pass', async function (this: AppWorld, days: number) {
await this.control.clock.advance(`P${days}D`);
});
Then('every charge that reached the provider carries the same idempotency key',
async function (this: AppWorld) {
const keys = this.mock.pg.received('POST /charges')
.map((call) => call.headers['idempotency-key']);
expect(new Set(keys).size).toBe(1); // however many retries, never a double charge
});
/__test/clock,
a Go rewrite has to serve the same window. That sounds like a cost but it is a gain: "can time be moved
forward?" is what decides whether a system is testable at all, and declaring it means the port cannot
lose that property.
13 The black-box boundary
An explicit line about what may and may not be touched.
| Action | Verdict | Reason |
|---|---|---|
| HTTP · GraphQL · gRPC · WS calls | allowed | The public contract — what a rewrite must preserve |
| Running the CLI, asserting on exit code and stdout | allowed | A window users actually use |
| Observing published events and messages | allowed | Side effects visible from outside |
| Driving the browser, locating by role or label | allowed | Exactly how a person uses the screen |
| Capturing screenshots, comparing approved baselines | allowed | Thresholds are declared, so it is deterministic |
| Programming mock infra and asserting on its received calls | allowed | Belay started those; it is still the SUT's observable behaviour |
Using declared control surfaces (/__test/*) | allowed | Part of the contract; a port implements it too |
| Reading or writing the database against its internal schema | never | Breaks on every structural change, and cannot survive a port at all |
Importing src/ or calling internal functions | never | Void the moment the language changes |
| Locating elements by CSS class or component instance | never | Breaks on a style change; wiped out by a framework swap |
exec into the SUT container, or setting its clock directly | never | If you need the control, declare it as a window |
| Asserting by matching log strings | never | Logs are not a contract. If you need to observe it, promote it to an interface |
| Letting model interpretation gate on its own | avoid | A non-deterministic verdict undermines the premise that results are trustworthy |
Fixed sleep to line up timing | avoid | The main cause of flakiness. Wait on a condition instead |
| Sharing state between scenarios, cookies included | avoid | Order dependence destroys confidence in the result |
There is one grey area worth naming: test-only interfaces. If a state genuinely cannot be
produced from outside — moving time, seeding fixtures — do not drill inward. Have the SUT serve an
explicit, documented window like /__test/clock. A port has to implement the same window,
so it stays part of the contract rather than becoming an exception to it.
14 Scenario layers
Directories are layers, layers are execution order — and the grammar you read failures with.
With layers, a failure is already a diagnosis. If L1 passes and L2 breaks, the individual capabilities are
fine and the wiring is not — and when you hand it to an agent, that information goes with it. For fast
feedback, cut the run: belay run --upto L1.
15 Porting and rewrites
The real reason Belay exists.
If the scenarios live outside src/ and speak only through public interfaces, then
src/ is a replaceable part. Rewriting an Express prototype in Kotlin and Spring, swapping REST
for GraphQL, moving a React screen to Svelte — the procedure is the same.
- Take a baseline. Confirm every layer passes with the current
src/and keep the report. A spec that does not pass cannot be a standard. - Swap the target. Put the new implementation in
src/and change onlybuildandstart.surfacesandtest/are not touched at all. - Light the layers up in order. Get L0 green, then climb. Each layer passing is progress, and the remaining failures are the remaining work.
- Call it done. Every layer green means the port is complete. Not "I think I moved everything" — a confirmed fact.
test/, it is usually one of two things: the scenario was
secretly coupled to the old implementation (fix the scenario), or the contract genuinely changed (fix the
spec, with agreement). Either way, never quietly lower a scenario to make it pass — that is
the same as leaving slack in the rope.
What changes when the surface changes
| Kind of change | What moves and what stays |
|---|---|
| Language or framework Express → Spring |
Only build/start. surfaces and test/ stay as they are. The purest form of a port. |
| Protocol REST → GraphQL |
The contract itself changes, so the driver calls in the steps change. But the .feature sentences must not — if the scenarios move, this is a feature change, not a port. |
| Front-end framework React → Svelte |
If roles, labels and data-testid survive, so do the steps. Needing to change a locator means the scenario was coupled to the framework. |
| Redesign | Structural assertions hold; visual baselines are invalidated. Renewing a baseline is an explicit, reviewed approval — automatic renewal erases the safety net. |
The same structure works when the work is handed to an agent. Give it write access to src/ and
read-only access to test/, then let it loop on belay run. It cannot rewrite its own
grading criteria, so the only thing that can converge is the implementation.
test/ read-only. It converged — 50/50 in
~14 runs and ~25 minutes, three undocumented behaviors reproduced exactly because the failure list demanded
them, grading criteria untouched. One caveat came out of the run: a small model stops early with failures
remaining. The failure list supplies direction; keeping the loop running is the job of whoever
holds the rope.
16 Results and trust
That it passed matters less than that the pass can be believed.
Evidence from every run
Step timeline, protocol transcripts, screenshots and visual diffs, process logs, and the DOM and console at the point of failure. You should not have to re-run anything to see what happened.
Non-deterministic verdicts kept apart
Visual interpretation is tallied separately as warnings, never mixed into pass/fail. The model's own words and the image behind them are kept, so a person can judge.
Flaky counts as failing
Retries are not used to manufacture green. An unstable scenario is quarantined and fixed — a safety net that wobbles is worse than none.
Undefined steps do not stay quiet
Steps with no definition and skipped scenarios are named in the result and subtracted from coverage. A silent pass is the most dangerous outcome there is.
Machine-readable output
junit.xml and json drop straight into CI gates and agent loops. The exit code alone should be enough to decide.
$ belay run
▸ infra postgres ✓ 1.8s kafka ✓ 4.1s payment-gateway ✓ 0.6s
▸ build belay-sut ✓ 24.1s
▸ ready http://localhost:8080/health ✓ 3.2s
▸ surfaces http · graphql · grpc · ws · browser(chromium) ✓
L0-smoke 2 scenarios 2 passed
L1-account 14 scenarios 14 passed
L1-catalog 6 scenarios 6 passed graphql
L2-order 9 scenarios 8 passed 1 failed grpc · ws
L2-ui-checkout 7 scenarios 7 passed browser · ⚠ 1 visual warning
L3-cross 4 scenarios 4 passed
✗ L2-order/checkout.feature:31 reserving more than available is refused
Then the gRPC status is "FAILED_PRECONDITION"
expected FAILED_PRECONDITION, received INTERNAL
→ .belay/reports/2026-07-29T20-14/L2-order-31/
⚠ L2-ui-checkout/cart.feature:24 [visual interpretation · not a gate]
claim: "no price is clipped or overlapping" — does not hold (confidence 0.71)
"on the mobile viewport the total overflows its container"
→ cart-filled.mobile.png · review, then promote to an assertion or a baseline
▸ teardown keepOnFailure=true — environment kept (belay down to clean up)
41/42 passed · 1 warning · 2m 38s · exit 1
17 Where this is verified
A design document that runs ahead of the code betrays its first reader. Here is which example proves which claim.
examples/ is a regression suite, not a gallery.
npm run examples runs all of them exactly as a user would, and CI treats a failure there as a
broken build.
| Example | Language | What it proves |
|---|---|---|
01-http-todo | English | In-process stubs without Docker · real SMTP capture · fault injection (500 · latency · partition) · clock control · waiting on async side effects |
02-porting | English | The same scenarios pass against both a Node and a Python implementation. The central claim of this document |
03-graphql-ws | English | GraphQL and WebSocket woven into one flow · waitFor/quietFor · refusing to call partial success a success |
04-grpc | English | Unary calls · transport code and domain code kept apart · idempotent replay · server streaming |
05-browser | English | Semantic locators · approved baseline comparison · per-viewport outlines · console error observation |
What is not implemented yet
What does not work — or deliberately never will — is named here rather than left to be discovered.
- The queue (Kafka) driver — implemented in 1.0 (
surfaces.queue, optional peerkafkajs): publish with keys and headers, latest-offset subscriptions with the samewaitFor/quietFor/markreplay contract as WebSocket, automatic topic creation via the admin API. Field-verified against a live single-node Kafka (apache/kafka 3.8, KRaft) — produce→consume roundtrip, key/header integrity, latest-offset semantics, silence assertions — and its buffering semantics are pinned by unit tests against the declared client seam. Like the Docker infra path, it is not continuously verified: the examples deliberately run without Docker, so no example exercises a live broker on every run. - The Docker infra path — the code to start containers via
image:exists and has been field-verified once (a Postgres container carried the RealWorld showcase, macOS/colima), but every example still runs without Docker, so it is not continuously verified. Prefercommand:,stub:andmail:. Since 1.0,ports: ['15432:5432']publishes host ≠ container mappings, and a readiness command can reach the generated container by name —ready: { command: 'docker exec {container} pg_isready' }— which is the honest probe behind a VM port-forwarder, where TCP accepts before the service does.
Windows
Supported natively (it used to be refused). Teardown kills the whole process tree with taskkill /T where POSIX kills the process group, and belay down additionally hunts down anything still holding a declared port. Two platform notes: stop signals are a POSIX concept — on Windows every stop is forceful, so an app that needs a graceful-shutdown window will not get one there; and --link symlink falls back to a directory junction when symlinks would need Developer Mode. WSL2 remains fully supported.
18 Glossary
The words used throughout.
| Term | Meaning |
|---|---|
| Belay | In climbing, holding the rope so a falling partner stops. The name of this framework and its job. |
| SUT | System under test — what sits in src/. Belay knows nothing of its internals. |
| Surface | A public window tests may knock on: HTTP, CLI, a queue. Only what is declared exists. |
| Feature | An executable specification written in Gherkin: the spec a person reads and the check a machine runs. |
| Step definition | The TypeScript that translates a Gherkin sentence into a driver call. Kept thin. |
| Driver | An adapter wrapping one surface. Carries no domain logic. |
| Exchange | The normalised result a driver returns. The same shape across protocols, which is what makes shared steps possible. |
| Semantic locator | Finding an element by role, label or accessible name. Survives a framework change. |
| Visual baseline | An approved reference screenshot. Committed, and renewed only by explicit, reviewed approval. |
| Visual interpretation | A model's judgement of a capture. Non-deterministic, so a warning rather than a gate by default. |
| Control plane | Declared surfaces for state you cannot create from outside: clock, seed, fault injection. |
| Mock infra | Test doubles for what the SUT depends on. Belay started them, so scenarios may program them freely. |
| Layer | Scenario layers L0–L3. Execution order, and the grammar for reading failures. |
| Baseline (run) | The all-layers-green record taken before a rewrite. The reference a port is measured against. |