Troubleshooting

On this page you find the problem you are looking at, learn why it happens, and fix it. Each entry has the same three parts: what you see, why, and what to do.

Start with ds machine ready. It proves the machine works, one line per proof, and every failing line names its fix. A full run takes a few minutes, because it starts apps and runs a landing check; ds machine ready --quick starts no app and calls no model. A line marked held (another session or check was using what that proof needed), you (a step only you can take) or note never stops the verdict being READY.

• Your AWS sign-in has lapsed
• Docker is not answering
• A credential is missing
• A checkout without its demo data
• AWS refuses the agent machine identity
• Another ds verify run is going
• A stopped setup or ready run left a checkout reserved
• Setup will not edit your shell startup file
• Setup stopped: a new terminal would run another Node
• The stage step is blocked
• Your stage's database is behind your code
• Claude in Chrome is offered but not enabled
• Many tests time out at once
• ds pool or ds work says you do not hold the checkout
• The merge guard refused a command
• When the hourly health check reports a problem
• Where the logs are

Your AWS sign-in has lapsed

What you see

• In a Claude Code or Codex session, the check that runs when a session starts prints one line saying the AWS session for your profile is expired or missing, with the command to run.
• A ds command that needs AWS opens the sign-in in your browser when you run it at a terminal. From an agent's shell, where nobody can finish a sign-in, it stops instead and prints the command to run, aws sso login --profile <profile>.
• ds machine ready fails a line in its aws group, ending sign in: aws sso login --profile <profile>.
• ds secrets pull warns that it could not refresh from AWS, and writes from its local copy instead.

Why

On a person's machine, setup writes a self-renewing AWS sign-in (the sso-session form of an AWS profile). The AWS command line renews it by itself for as long as your AWS IAM Identity Center session lasts. When that session ends, you sign in once more.

Agent machines never sign in: they use certificates instead, and a sign-in can never fix a refused certificate. If the message talks about a machine identity, go to AWS refuses the agent machine identity.

What to do

1. Sign in with the profile the message names:

   aws sso login --profile <profile>
   

   
   Setup's profile is devstride-dev unless you already had one for the dev account, and setup's block in your shell startup file exports it as DEVSTRIDE_DEV_PROFILE, so aws sso login --profile "$DEVSTRIDE_DEV_PROFILE" also works. Inside a Claude Code session, type it with a leading ! (! aws sso login --profile <profile>) to run it from the prompt.
2. Re-run whatever failed. Nothing retries it for you.

Always use the command spelled out in full. Some machines have a shell helper called sso; it exists only where someone pasted it into their shell, and it is never the fix to give anyone.

A lapsed sign-in blocks only what reaches AWS. The local Docker app, ds pool, ds work and ds verify need no sign-in, and ds secrets pull keeps writing from its local copy until you sign in again.

Two cases a sign-in will not fix:

• The line says the profile is not configured at all. Run ./setup again: on a person's machine it writes the sign-in session and the dev profile to ~/.aws/config. On an agent machine, run ./setup --agent <name>, which enrols the machine and writes its profiles.
• Your sign-in lapses about every eight hours. Your profile is the older form that cannot renew itself. ./setup names the one-line change; the repository's README, "AWS SSO setup", shows both forms.

Docker is not answering

What you see

• The local app does not load, and ds pool status ends with Docker is not reachable — containers not surveyed.
• ds machine ready fails a checkout's database line, ending is Docker running?
• Tests and ds verify cannot reach the test databases. ds verify full says the test Postgres is not accepting connections.
• Setup waits for Docker Desktop, or blocks with Docker has <n> CPUs and <x> GB; the backend tests need at least 4 CPUs and 8 GB — raise them in Docker Desktop: Settings, Resources.

Why

Every checkout's app, its database and its test databases run in Docker. On a person's Mac, setup leaves Docker Desktop's own start-up settings as they are, so after a restart Docker may simply not be running. (On an agent machine, setup makes Docker Desktop start whenever someone signs in to the Mac.) The backend tests need at least 4 CPUs and 8 GB of memory given to Docker, which is why setup checks both.

What to do

1. Open Docker Desktop and wait until it says it is running. If it shows a welcome screen or asks you to sign in, finish or close it: signing in to Docker is not needed.
2. Start the apps again. ds worktree up main starts main's app. A wt1 or wt2 app starts when you take that checkout with ds pool checkout.
3. If setup reported too few CPUs or too little memory: in Docker Desktop, open Settings → Resources and give it at least 4 CPUs and 8 GB of memory. Run ./setup again; it re-checks Docker's CPUs and memory.

A credential is missing

What you see

• A command refuses for want of a key. For example ds seed or ds neon says SEED_TOKEN or NEON_API_KEY is not in the agent credentials file, and names the file.
• ds machine ready fails a line in its credentials group: a checkout's .env is missing keys the bundles deliver, or the agent credentials file lacks a DevStride key. Each line ends with what to run.
• In its aws group, the secrets line says nothing has been pulled yet, or that a bundle has a newer version: run: ds secrets pull.
• At the end of ./setup, under When you can, a note says SEED_TOKEN and NEON_API_KEY are not on this machine yet.

Why

Keys are never copied by hand. They live in three bundles in the dev AWS account's Secrets Manager: a shared bundle every machine gets, one machine bundle with a single machine's own stage settings, and an agent bundle with the agents' own service keys. ds secrets pull writes the shared and machine bundles into every checkout's .env, and the agent bundle only into ~/.config/devstride/agent.env, never into a .env.

A key is missing for one of two reasons: this machine has not pulled since the key was added, or nobody has stored it yet.

Today's decision (owner, 2026-10-05): the keys are shared, each with full access, to get everything working. Narrower access per person or per machine comes later.

What to do

1. Pull the latest keys. This needs a working AWS sign-in (see Your AWS sign-in has lapsed):

   ds secrets pull
   

   
   It rewrites every checkout's .env completely, so never edit a .env by hand: the next pull overwrites it.
2. Prove it with ds machine ready (or ds machine ready --quick). Each credentials line names whatever is still missing.
3. If the key is in no bundle yet, the owner creates it with the service, then stores it. The value goes in on stdin, so it never lands in shell history:

   pbpaste | ds secrets set agent <KEY>
   

   
   Then every machine runs ds secrets pull.

The Seed and Neon keys. SEED_TOKEN (for ds seed deploy, which redeploys a chosen commit) and NEON_API_KEY (for ds neon, which runs the command-line tool of Neon, our Postgres host) are created by an admin, in Seed's and Neon's organisation settings (For admins). Until they exist, those two commands refuse. Your machine's AWS stage also waits on NEON_API_KEY, so setup lists the stage under Needs you until it arrives; nothing else waits on either key. Seed needs no key for ordinary deploys: a merge to develop or master deploys by itself.

Every tool's credential (where it comes from, which bundle holds it, who creates and rotates it, how ds machine ready proves it) is in the repository's credentials map, .claude/skills/ds-credentials/SKILL.md. Credentials and access walks through it.

A checkout without its demo data

What you see

• ds pool status says a checkout's data is not recorded (its data could be anything).
• At the end of ./setup, under When you can:
  • a demo-data record for wt1 (or wt2) line beginning not recorded: its database already exists, so the hourly check leaves its data alone; or
  • a demo data in wt1 (or wt2) line saying why setup left it as it is: another session holds it, it is on a branch, it has uncommitted changes, or it holds organisations besides the demo data.
• ds machine ready fails a checkout's demo sign-in, saying acme.devstride could not sign in.
• ds fleet status marks the checkout unrecorded.

Why

Setup builds the demo data once, keeps it as the machine's snapshot, and copies it into each checkout. It only replaces a database it knows is safe to replace: in a checkout nobody is using, holding nothing or only the demo data.

A checkout whose database already existed before setup could record it is not recorded. Its data could be anything, a customer's organisation copied in by hand for example, so nothing resets it on its own: not setup, not the hourly check (ds machine doctor, which keeps idle checkouts current on an agent machine). It waits until a person has looked.

What to do, for wt1 or wt2

1. Check that the checkout really holds nothing but demo data you can lose. A reset replaces its whole database.
2. When it is free (nobody holds it, it is not on a branch, it has no uncommitted changes), run the reset that setup printed, under your session's name:

   export DEVSTRIDE_SESSION="<your name>"
   ds pool checkout wt1 --purpose "demo data"
   # in a plain terminal, now run the export DEVSTRIDE_LEASE_TOKEN=… line it printed
   ds pool reset wt1
   ds pool checkin wt1
   

   
   The reset copies the snapshot in, re-creates the demo logins and records the checkout as holding demo data, so from then on setup and the hourly check keep it current.
3. If setup left it because someone was using it, let that work finish and the checkout be handed back. Then run the reset above, or run ./setup again, which fills an empty checkout from the snapshot.

If the reset says there is no golden template yet, the machine has no snapshot. Run ./setup again: it builds one in a free wt1 or wt2, which takes a few minutes.

For main. Setup fills main's database only while it holds nothing else and nobody holds main, and it never moves main's branch or stops its app. When main needs the demo organisation back, rebuild just that organisation in place:

ds worktree cli main golden build     # rebuilds only the Acme demo organisation; other data is untouched
ds worktree cli main golden cognito   # or: only re-create the demo logins

This rebuild is what setup, ds machine ready and the hourly check print for main. A reservation would not do: taking main needs --i-know-main-is-free, and handing it back stops its app.

AWS refuses the agent machine identity

What you see

• The check at session start says the AWS machine identity for the profile was refused, and that a login will not fix it, because it is certificate-based.
• ds stops with the same message instead of opening a sign-in.
• ./setup --agent <name> blocks with AWS refused <name>'s identity (revoked, expired, or its key is gone) — setup never replaces an identity by itself; re-enrol on purpose: ds machine enroll <name> --force, then run setup again.
• ds machine ready fails its aws lines: the machine's certificate identity was refused or has expired.

Why

An agent machine (one that runs sessions with nobody at the keyboard) cannot sign in through a browser, so it has its own certificates: one for the dev AWS account and one for production. AWS exchanges them for short sessions named after the machine. They last a year.

AWS refuses them when the machine was revoked (ds machine revoke), when the certificate has expired, or when the machine's key file is gone. No sign-in can fix that, so nothing offers one.

What to do

1. If the machine was revoked on purpose (retired, lost or stolen), stop here: it should stay shut off.
2. Otherwise give it a new identity. This is also the yearly renewal:

   ds machine enroll <name> --force
   ./setup --agent <name>
   

   
   The first command makes new keys, and removes the old certificates and the machine's two AWS profiles. Setup then asks for one AWS approval in the browser, signs and installs the new certificates, and checks that both accounts answer as this machine. The machine has no AWS access from the first command until setup finishes, so do it while no session needs AWS.
3. Check it:

   aws sts get-caller-identity --profile devstride-machine-<name>-dev
   

   
   The Arn it prints (the AWS name of the identity in use) ends in assumed-role/devstride-machine-identity/<name>.

When the approver cannot sign: the administrator route. The one approval must come from someone whose AWS sign-in can sign with the management account (where the certificate authority lives). If yours cannot, setup says so, and names this route instead:

1. Setup has already made the machine's two certificate requests, dev.csr.pem and prod.csr.pem, in ~/.devstride/machine-identity/<name>/ on the agent machine. They hold no secrets: copy them to the administrator. The machine's private keys (dev.key, prod.key) never leave it.
2. The administrator, in their own DevStride checkout and signed in to the management account, signs both:

   ds machine sign <name> --account dev --csr <full path to dev.csr.pem> --out dev.signed.pem
   ds machine sign <name> --account prod --csr <full path to prod.csr.pem> --out prod.signed.pem
   

   
   ./ds runs from the top of the checkout, so give the requests' full paths; the certificates are written to the top of that checkout. It signs with the devstride-mgmt profile unless --mgmt-profile (or DEVSTRIDE_MGMT_PROFILE) names another. The certificate names the machine and account given on the command line, whatever the request says.
3. Save the two certificates on the agent machine as ~/.devstride/machine-identity/<name>/dev.signed.pem and prod.signed.pem, then run ./setup --agent <name> again. It installs them with no sign-in.

The repository's ds-machine-identity skill is the full runbook, including how to revoke a machine.

Another ds verify run is going

What you see

• ds verify land or ds verify full waits, and about once a minute prints a line ending — one deep check per machine at a time, naming the folder, process and start time of the run it is waiting on.
• ds work land seems to hang at its local check, which is that same command. Its waiting line goes to .ds/work-land.log, not to the screen.
• ds machine ready marks its landing check held: another ds verify run is going, then run ds machine ready again once it has finished.

Why

A machine runs one deep check at a time. Two full test runs on one machine starve each other of CPU, and the timeouts that follow look exactly like broken code. So every ds verify run takes a machine-wide lock, ~/.cache/devstride/verify.lock, and the next one waits its turn.

What to do

Wait: your run starts as soon as the other finishes. The waiting line tells you whose run it is. There is nothing to clear by hand: a run that died leaves its lock behind, and the next run reclaims it by itself and says so. For ds machine ready, run it again once the other run has finished.

A stopped setup or ready run left a checkout reserved

What you see

• ds pool status shows a checkout held by machine-setup, with a purpose such as machine setup demo data pid=<n> or machine ready pid=<n>. ds pool checkout on it answers BUSY.
• When it was stopped, setup printed Stopped while setup held <checkout>: its lease stays, because a copy may still be finishing inside Docker.
• ds machine ready marks that checkout held, saying it was left by a stopped ./setup or ds machine ready run and that the next full ds machine ready or ./setup takes it back.

Why

Setup and ds machine ready reserve a checkout while they copy demo data into it or start its app. If they are stopped part-way (Ctrl-C, a closed terminal), they deliberately keep the reservation: a copy may still be finishing inside Docker, and handing the checkout over then would give someone a database that is still being rewritten.

What to do

Nothing by hand. Once the stopped process has gone, the next full run takes the reservation back and stops any app it left running: ./setup, a full ds machine ready (not --quick), or on an agent machine the hourly check. It says took back the reservation a stopped setup run left on <checkout>. Never delete the reservation yourself.

The hourly check (ds machine doctor) keeps its own reservation the same way when it is stopped part-way. It prints the exact command that frees the checkout sooner, once nothing is running there, and its next scheduled run takes the checkout back anyway.

Setup will not edit your shell startup file

What you see

./setup lists a shell startup files line under Needs you:

setup will not edit <file> (it has a "# >>> devstride machine setup >>>" or "# <<< devstride machine setup <<<" line without its pair, or more than one block) — leave one pair of its marker lines, or none, then run setup again

Why

Setup keeps its lines (ds on your PATH, Node, this machine's name and AWS profile) in one block between two marker lines in your shell's startup file, and rewrites only what is between them. When a marker is missing, or there are two blocks, it cannot tell where its block ends: pairing a stray start line with a later end line would delete your own lines in between. So it changes nothing and asks you.

The file depends on your shell: for zsh, ~/.zshrc, plus a second small block in ~/.zshenv, which every zsh reads, scripts included; for bash on a Mac, ~/.bash_profile (or ~/.bash_login or ~/.profile, whichever bash reads); for bash on Linux, ~/.bashrc. The message names the file.

What to do

1. Open the file it names and find the lines # >>> devstride machine setup >>> and # <<< devstride machine setup <<<.
2. Leave exactly one start line followed by one end line around setup's lines, or delete setup's lines together with both markers.
3. Run ./setup again. It rewrites everything between the pair, or adds a fresh block when there is none.

Before its first change to a startup file, setup keeps a copy of it beside the original with .before-devstride-setup added, for example ~/.zshrc.before-devstride-setup. Compare against it to tell your own lines from setup's. Keep your own lines outside the block: setup rewrites the block on every run.

Setup stopped: a new terminal would run another Node

What you see

./setup lists a shell startup files line like this:

setup's shell blocks would make a new terminal run another node (node: v22.11.0 → v24.3.0); they were taken back out and your startup files are as they were — this is a setup defect: tell the team, with this line

Why

After writing its block into your shell's startup files, setup opens a fresh login shell and checks which Node and pnpm it finds. Setup's block must never switch your terminal to a different Node or pnpm than the one it ran before (from nvm, fnm, Homebrew or another version manager), because that could break other projects on your machine. When it would, setup takes its block back out at once, so nothing about your terminal has changed.

What to do

1. Send the team the line setup printed: it names the Node and pnpm before and after, which is what is needed to fix setup.
2. Meanwhile, your terminal is exactly as it was. Everything else setup did stays done; only ds on your PATH and the machine's name in new terminals are missing. Use ./ds from inside a checkout until the fix lands.
3. If you want to compare by hand, setup kept a copy of every startup file it touched, ending .before-devstride-setup (for example ~/.zshrc.before-devstride-setup).

The stage step is blocked

What you see

./setup lists the stage under Needs you, or ds machine ready fails a stage: line. The line says which of these it is:

• this machine has no Neon key: the team's NEON_API_KEY is not on this machine;
• holds a database but names no stage: a stage made by hand: this machine had a stage before setup made them;
• could not look at this machine's stage: the AWS sign-in lapsed or the look failed;
• a failed first deploy, with the deploy's last lines.

Why

Your stage's database is its own Neon branch, made with the team's Neon key, and its first deploy runs from a pool checkout. Setup builds the stage only when everything it needs is in place, and never rebuilds a stage that already exists.

What to do

• No Neon key: an admin adds it once for the whole team (For admins). Then run ds secrets pull and ./setup again.
• A stage made by hand: adopt it once, never rebuilt. Take wt1, then name the stage: ds stage create --stage <its name> --member wt1.
• AWS sign-in lapsed: see Your AWS sign-in has lapsed, then run ./setup again.
• A failed first deploy: read .ds/stage-deploy.log in the checkout it deployed from (setup names it). Fix what it shows, or send it to the team, then run ./setup again: it finishes the stage rather than starting over. ds stage create --member wt1 --dry-run shows what is left to do.

Your stage's database is behind your code

What you see

ds machine ready shows a note on stage: database, for example 3 migration(s) behind main's code.

Why

Setup never redeploys your stage because develop moved on. Your checkouts' code moves forward; the stage's database stays where it was until you migrate it.

What to do

From the checkout whose code you will run against the stage, run the command the note gives, for example:

DEVSTRIDE_STAGE=<stage> DEVSTRIDE_REGION=us-east-1 ./ds -b migrations run

Claude in Chrome is offered but not enabled

What you see

Setup's When you can list, or a ds machine ready note, says Claude in Chrome is offered but not enabled yet.

Why

Setup asks Chrome to offer the Claude extension, but only you can turn an extension on and sign in to it. Agents do their browser work through it, so until it is enabled they hand browser steps back to you.

What to do

Open Chrome. It shows a notice that a new extension, Claude, was added: click Enable. Then open the extension and sign in to Claude. If you dismissed the notice, open chrome://extensions and switch Claude on there. On an agent machine, do this in its DevStride Agent Chrome profile.

Many tests time out at once

What you see

Dozens of test files fail with Test timed out or Hook timed out, or a run takes several times longer than usual. Two signs that the code is not to blame: a different set of files fails on each run, and the failing tests were skipped with no assertions after a setup step timed out, rather than failing an assertion.

Why

Three causes look exactly like broken code:

1. Another checkout on this machine is running a test suite at the same time. Each checkout has its own test databases, so the two runs cannot corrupt each other, but they starve each other of CPU.
2. This checkout's test databases have degraded, after a long uptime or a stopped run that left its databases behind.
3. The test databases restarted seconds ago and Postgres is not accepting connections yet. The failures then cluster at the very start of the run, all in the setup step.

What to do

1. Rule out the first cause before anything else. Run docker ps and look for another checkout's devstride-*-drizzle-tests container with a run in progress, and ask the session using it. ds verify already runs one deep check per machine; when two runs genuinely must overlap, cap each with --maxWorkers=7. A high load average alone proves nothing.
2. Restart this checkout's two test databases. ds worktree ls names the pair:

   docker restart devstride-<slug>-drizzle-tests devstride-<slug>-dynamodb-tests
   

3. Wait until Postgres answers. Use a readiness check, never a fixed pause:

   until docker exec devstride-<slug>-drizzle-tests pg_isready -U postgres; do sleep 1; done
   

4. Run again. If it still fails, run the same files on a commit you know was good. Failing the same way there means the environment; passing there means your change.

Two traps hide the real result:

• Piping a test run through tail or grep reports the pipe's exit code, so a failed run reads as a success. Write the output to a file instead, from the backend folder: pnpm test:suite:ci:non-golden > run.log 2>&1; echo "GATE_EXIT_CODE=$?".
• After Docker or the Mac crashes, the test Postgres may be damaged: for speed it runs without crash-safety flushes. If the suite cannot connect, or reports missing files, remove this checkout's test Postgres with docker rm -f devstride-<slug>-drizzle-tests; the next run recreates it.

ds verify land already re-runs, once, up to five files that timed out during setup when no test failed. The full procedure, with measured examples, is in the repository's testing guide: Diagnosing a mass test failure. Running Tests Per Instance explains the per-checkout test databases.

ds pool or ds work says you do not hold the checkout

What you see

• ds pool reset, ds pool checkin, ds work start or ds work land refuses, saying the checkout is leased under your session name but by a different session, that you do not hold it, and that a lease taken from a plain terminal needs the DEVSTRIDE_LEASE_TOKEN that ds pool checkout printed there.
• Or ds work says this checkout is not leased to you, and names the ds pool checkout command to run.

Why

A lease records which session took it, not just the name: two sessions can share one name. A Claude Code or Codex session proves it is the holder with its own session id. A plain terminal has no such id, so ds pool checkout handed it a one-time token instead.

What to do

• In the plain terminal that took the lease, run the export DEVSTRIDE_LEASE_TOKEN=… line ds pool checkout printed, then try again. Run later commands in that same terminal.
• If you never took the lease, take it: ds pool checkout <checkout> --purpose <item>. If that says BUSY, another session holds it: ask that session, or take another checkout.
• A lease taken before leases recorded their holder is refused by these commands. Hand it back by hand only if you are sure it is yours; otherwise it is the owner's to clear.

The merge guard refused a command

What you see

A git push, git merge, gh pr merge or gh api command is refused before it runs, with a one-sentence reason from the merge guard.

Why

Every Claude Code and Codex session runs the merge guard before each shell command. It refuses the commands that would skip review: a push to develop or master, a forced or deleting push to a release branch, a merge into develop from a branch the merge-path rule does not admit, a merge into master from anything but a release or hotfix branch, and writes to GitHub's branch rules. Develop has the full list.

What to do

• Take the normal route: ds work land (or /devstride:build-item) for a Story or one-off, a pull request for anything bound for develop, and /devstride:release for production.
• If you think the refusal is wrong, ask the rule for its verdict and reason:

  ds merge-path check --head <branch> --base develop --labels <label1,label2>
  

  
  If the rule itself is wrong, the fix is the rule (mergePath in .claude/ds-config.json), never the guard's switch.

When the hourly health check reports a problem

The health check (ds machine doctor, run hourly on an agent machine and inside every ds pool checkout) fixes what it may and reports the rest. What its messages mean:

Where the logs are

Paths starting with .ds/ are inside a checkout. Setup's and ds machine ready's logs are always in the main checkout, ~/dev/devstride.

When you ask for help with a failed setup step, send the line setup printed for that step, .ds/machine-setup.log, and for the demo data .ds/machine-setup-data.log.

Next: back to the Overview