My password manager ran out of yeses

At 4:19 on a Friday morning, every script on my home server that needed a password stopped getting one.

Nothing crashed. The containers already running carried on, because they’d read their secrets when they started. But anything that started fresh and asked for a secret got the same answer: Too many requests. Over the next 19 hours that answer appeared 912 times.

One allowance for everything

I don’t keep passwords and API keys in files on the server. The files hold references to 1Password, and each script looks up the real values when it runs. It’s a good habit. It means a leaked config file leaks nothing.

What I hadn’t taken on board was the budget. My 1Password account allows 1,000 requests a day, and that one allowance is shared by everything: every scheduled job, every deployment, every Claude session that starts up and needs its tools’ keys. When it’s gone, it’s gone for all of them at once.

Two jobs spent it:

• A client’s email job checked a mailbox every two minutes. On every run it asked 1Password for my entire secrets file, about 180 references, to use four of them.
• Rulebook’s answer collector had run every ten minutes. On 9 September I sped it up to every two. The same afternoon, the email job gained one more lookup per run, for the token that reports its heartbeat to my uptime monitor.

Thirty-eight hours after that change, the first request was refused.

Why it took 38 hours

On paper it should have gone in hours. 1Password’s command-line tool keeps a local cache, though, and I measured it once the allowance came back: a lookup the cache can answer costs nothing against the allowance, and one it can’t costs two requests. A cold read of the whole secrets file, which points at 68 separate items, costs about 136.

So the drain was never the number of lookups. It was cache misses. The email job had already done about 4,600 full-file lookups over the previous week without a single refusal. It was the extra traffic from 9 September that tipped it over. I never measured the miss rate itself, so the exact arithmetic of 38 hours is still an estimate.

It wasn’t the AI bill either. The email job had spent $1.16 on model calls across 30 emails. The expensive thing on the box was a free password lookup, repeated.

What went down

Everything that reads a secret at start:

• The email job and Rulebook’s collector, which had caused it.
• Both of my paper-trading scans, which missed their morning runs.
• The heartbeats that jobs send to my uptime monitor, because each one needs a token from 1Password to report in. The monitor could see jobs going quiet, but not why.

The fix: ask once

The rule now is to resolve secrets once per process, never per run.

• The two-minute jobs became long-running loops. Each reads its secrets once when it starts, keeps them in memory, and restarts with a back-off so a broken lookup can’t hammer the account.
• Each asks only for the references it needs, never the whole file.
• Heartbeat tokens are read once a day into a locked-down local cache.
• A new check reads the allowance every 30 minutes and raises an alarm when less than 30% is left.
• Before any job is added or sped up, its lookups per day get added to the budget first.

The fix was built and tested that afternoon. It couldn’t go live until the allowance came back, which was hours away, so Claude booked the cutover for 10:47 pm.

The schedule that wasn’t

It booked it with Claude Code’s built-in scheduler, three times in one afternoon, all for 10:47 pm.

That scheduler lives in the memory of the Claude session. I work from a Windows PC, in a VS Code terminal connected to the server over SSH. When that terminal closes, the session goes with it, and so does everything it had scheduled.

At some point after that, my terminal closed. At 10:47 pm nothing happened, and nothing reported that nothing had happened. At 11:24 pm the allowance came back and the old two-minute collector started up again, which is exactly what the booking was there to prevent.

I found out at 7:28 the next morning, when I asked whether the timers had run. They hadn’t. The cutover went in by hand at 7:36.

Anything that has to happen while nobody’s watching now goes in a real scheduler on the server (a systemd timer), or the session runs inside tmux so it survives the terminal. A schedule that lives inside a chat session isn’t a schedule.

The watchdog that was mostly asleep

While we were in the logs, Claude looked at claude-watchdog, a script that runs every five minutes and kills stuck Claude processes. Its log was a wall of skipped reason=lock_active, and the first line was from March. Claude told me it had skipped every run since March.

That was wrong, and the truth was stranger.

The log wasn’t a log of runs. The watchdog writes a line only when it kills something or skips. A clean run writes nothing. The March skip was a one-off; the next one wasn’t until 8 August. A quiet log proves nothing about whether a script is running.

But from 8 August it really had been skipping: 9,466 of the 10,052 five-minute slots between then and the fix, 94%. Apart from one two-day stretch in August, a single run got through once every day or two. Its last kill had been on 9 September.

The cause was its own heartbeat. The watchdog takes a lock so that two copies can’t overlap, then asks 1Password for the token it uses to report that it’s alive. That lookup can start 1Password’s background helper, which inherits the watchdog’s open lock file and keeps running after the watchdog exits. The lock stays held until the helper quits, sometimes hours later, and every watchdog run in between sees the lock taken and gives up. The system’s own lock table named the owner as a process that no longer existed.

The call meant to prove the watchdog was alive was what stopped it running. It now reads that token from the daily cache, so it never starts the helper while it’s holding the lock.

The parts I’d reuse

• A shared allowance is a budget. Add up a job’s calls per day before you add the job, and before you speed one up.
• Read secrets once per process, and only the ones you need.
• Watch the allowance, not just the jobs. The one command that shows how much is left would have explained the 1Password half of this straight away.
• A schedule that lives in a chat session isn’t a schedule.
• A log of exceptions isn’t a log of runs. Before saying a job never ran, find out what it writes when it succeeds.
• Close a lock before calling anything that might leave a process behind. A background helper that inherits the lock keeps it.