[fix] release: stop the snapshot job starving the release job
Publish snapshot / snapshot (push) Successful in 1m5s
Publish snapshot / snapshot (push) Successful in 1m5s
Pushing a release pushes the branch and then the tags seconds apart. publish-snapshot.yml keys its concurrency group on the branch and release.yml keys its own on the tag, so the two never gate each other — on a single runner they race for it and the branch push always wins. On 1.0.1 the snapshot job wedged extracting a layer of the runner image, the release job never started, release.mjs gave up after its 20-minute wait, and two of three tags were left unpushed. The report still printed a bold "Done" above an empty shipped list, so it read as a success. - publish-snapshot.yml skips commits whose message starts with "release:". A snapshot of a release commit is the same tree the tag is about to publish properly, so skipping costs nothing and removes the race. - waitForRelease() offers to keep waiting instead of giving up. No timeout value survives a wedged runner, so the real choice is between asking and making the operator finish the release by hand. --yes and a non-interactive run still give up; the latter matters because confirm() answers with its default without a terminal, which would extend the deadline forever. - The final header says "Blocked" when it is, and labels the packages that did ship before the blockage. - --wait-timeout defaults to 2400s rather than 1200s. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DCzYTAm9QUhvLNr2EpdagJ
This commit is contained in:
co-authored by
Claude Opus 5
parent
eac44b637e
commit
2810d4491f
@@ -20,6 +20,17 @@ concurrency:
|
|||||||
|
|
||||||
jobs:
|
jobs:
|
||||||
snapshot:
|
snapshot:
|
||||||
|
# Not on a release commit. `scripts/release.mjs` pushes the branch and then
|
||||||
|
# the tags seconds apart, and this workflow's concurrency group is keyed on
|
||||||
|
# the branch while release.yml's is keyed on the tag — so the two never gate
|
||||||
|
# each other, they race for the runner, and the branch push always gets
|
||||||
|
# there first. Measured: a snapshot job that wedged pulling the runner image
|
||||||
|
# held the runner long enough for release.mjs to give up waiting on npmjs,
|
||||||
|
# leaving two of three tags unpushed.
|
||||||
|
#
|
||||||
|
# Skipping costs nothing. A snapshot of a release commit is the same tree
|
||||||
|
# the tag is about to publish properly, under a version nobody installs.
|
||||||
|
if: ${{ !startsWith(github.event.head_commit.message, 'release:') }}
|
||||||
runs-on: ubuntu-latest
|
runs-on: ubuntu-latest
|
||||||
|
|
||||||
env:
|
env:
|
||||||
|
|||||||
@@ -342,12 +342,15 @@ regardless of `--tag`; on Gitea it did not exist at all. Note that `npm view
|
|||||||
<name>` against a registry with no `latest` tag prints nothing and exits **0**,
|
<name>` against a registry with no `latest` tag prints nothing and exits **0**,
|
||||||
which is why this looked like a working lookup. (`npm view <name>@<version>`
|
which is why this looked like a working lookup. (`npm view <name>@<version>`
|
||||||
does exit 1 for a missing version, so the workflows' idempotency guards are
|
does exit 1 for a missing version, so the workflows' idempotency guards are
|
||||||
fine.) All four packages were reset to `0.5.0`; `1.0.0-alpha5` stays the
|
fine.) All four packages were reset to `0.5.0`, and `nopy`, `nopy-cubes` and
|
||||||
numerically highest version on npmjs, so install with an explicit `@latest`.
|
`nopy-cubes-core` have since gone out as `1.0.1` — the first release where
|
||||||
|
`latest` actually moved on both registries. `keyman` is still `0.7.0` and on
|
||||||
|
neither.
|
||||||
|
|
||||||
- Push to `main` → `publish-snapshot.yml` publishes every package to the Gitea
|
- Push to `main` → `publish-snapshot.yml` publishes every package to the Gitea
|
||||||
registry as `<version>-main.<run>.g<sha>` under the `main` dist-tag. The
|
registry as `<version>-main.<run>.g<sha>` under the `main` dist-tag. The
|
||||||
version is set on the runner with `npm pkg set` and never committed.
|
version is set on the runner with `npm pkg set` and never committed. It skips
|
||||||
|
commits whose message starts with `release:` — see *Runner contention* below.
|
||||||
- `git tag <dir>-v<version>` (e.g. `nopy-v1.2.0` — the directory under
|
- `git tag <dir>-v<version>` (e.g. `nopy-v1.2.0` — the directory under
|
||||||
`packages/`, not the npm name) → `release.yml` publishes to Gitea *and* npmjs.
|
`packages/`, not the npm name) → `release.yml` publishes to Gitea *and* npmjs.
|
||||||
The tag chooses the package, `package.json` supplies the version, and the run
|
The tag chooses the package, `package.json` supplies the version, and the run
|
||||||
@@ -400,12 +403,27 @@ gets `-vvv --debug`. Treat `docs/REFACTORING.md` as a plan, not a record.
|
|||||||
|
|
||||||
The publish lane has now run against the Gitea registry: all four packages are
|
The publish lane has now run against the Gitea registry: all four packages are
|
||||||
there under `@main`, and `pnpm run try:snapshot` installs them into a throwaway
|
there under `@main`, and `pnpm run try:snapshot` installs them into a throwaway
|
||||||
project with npm and runs the binary. The npmjs lane has only ever published
|
project with npm and runs the binary. The npmjs lane has published `nopy`,
|
||||||
`@bitsquare/nopy`; `keyman`, `nopy-cubes` and `nopy-cubes-core` have never been
|
`nopy-cubes` and `nopy-cubes-core` at `1.0.1`; `keyman` has never been released
|
||||||
released there. That used to be caught by the *check linked deps are released*
|
there. Ordering used to be enforced by the *check linked deps are released*
|
||||||
guard in `release.yml`; now it is `pnpm run release` that holds `nopy`'s tag back
|
guard in `release.yml`; now it is `pnpm run release` that holds `nopy`'s tag back
|
||||||
until `nopy-cubes` answers on npmjs.
|
until `nopy-cubes` answers on npmjs.
|
||||||
|
|
||||||
|
### Runner contention
|
||||||
|
|
||||||
|
`scripts/release.mjs` pushes the branch and then the tags seconds apart.
|
||||||
|
`publish-snapshot.yml` keys its concurrency group on the branch and `release.yml`
|
||||||
|
keys its own on the tag, so the two workflows never gate each other — on a
|
||||||
|
single runner they simply race for it, and the branch push always wins. The
|
||||||
|
1.0.1 release is what surfaced this: the snapshot job wedged extracting a layer
|
||||||
|
of `runner-images:ubuntu-latest`, the release job never started, `release.mjs`
|
||||||
|
gave up after its 20-minute wait, and two of the three tags were left unpushed
|
||||||
|
while the report still printed a bold **Done**. Three things changed as a
|
||||||
|
result — the snapshot job skips `release:` commits, the wait offers to keep
|
||||||
|
waiting rather than giving up (no timeout survives a wedged runner), and the
|
||||||
|
final header says **Blocked** when it is. The tags were pushed by hand
|
||||||
|
afterwards; all three packages are on npmjs.
|
||||||
|
|
||||||
Nothing checks that a bundle and the CLI reading it are compatible versions;
|
Nothing checks that a bundle and the CLI reading it are compatible versions;
|
||||||
`nopy.engines` was considered and deferred. `docs/CUBE-PACKAGES.md` is where all
|
`nopy.engines` was considered and deferred. `docs/CUBE-PACKAGES.md` is where all
|
||||||
of this came from and is now a record of what was built, including what differed
|
of this came from and is now a record of what was built, including what differed
|
||||||
|
|||||||
+38
-8
@@ -165,10 +165,21 @@ async function versionExists(name, version) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/** Polls until `name@version` resolves on npmjs, or the deadline passes. */
|
/**
|
||||||
async function waitForRelease(name, version, timeoutMs) {
|
* Polls until `name@version` resolves on npmjs, offering to keep waiting when
|
||||||
|
* the deadline passes.
|
||||||
|
*
|
||||||
|
* The offer is the point. A timeout here is far more often a runner that has
|
||||||
|
* not started the job than a release that failed, and no timeout value survives
|
||||||
|
* a runner that has wedged — so the choice is between asking and making the
|
||||||
|
* operator finish the release by hand. `--yes` and a non-interactive run give
|
||||||
|
* up instead, the latter because {@link confirm} answers with its default when
|
||||||
|
* there is no terminal, which here would extend the deadline forever.
|
||||||
|
*/
|
||||||
|
async function waitForRelease(name, version, timeoutMs, mayExtend) {
|
||||||
const started = Date.now();
|
const started = Date.now();
|
||||||
const label = `${name}@${version}`;
|
const label = `${name}@${version}`;
|
||||||
|
let deadline = started + timeoutMs;
|
||||||
process.stdout.write(` waiting for ${label} on npmjs `);
|
process.stdout.write(` waiting for ${label} on npmjs `);
|
||||||
|
|
||||||
for (;;) {
|
for (;;) {
|
||||||
@@ -177,9 +188,16 @@ async function waitForRelease(name, version, timeoutMs) {
|
|||||||
console.log(chalk.green(` published after ${seconds}s`));
|
console.log(chalk.green(` published after ${seconds}s`));
|
||||||
return true;
|
return true;
|
||||||
}
|
}
|
||||||
if (Date.now() - started > timeoutMs) {
|
if (Date.now() > deadline) {
|
||||||
console.log(chalk.red(' timed out'));
|
console.log(chalk.red(' timed out'));
|
||||||
return false;
|
if (!mayExtend || !process.stdin.isTTY) return false;
|
||||||
|
console.log(
|
||||||
|
chalk.dim(' A queued or wedged runner looks exactly like this — check the run.')
|
||||||
|
);
|
||||||
|
if (!(await confirm(`Keep waiting for ${label}?`, true))) return false;
|
||||||
|
deadline = Date.now() + timeoutMs;
|
||||||
|
process.stdout.write(` waiting for ${label} on npmjs `);
|
||||||
|
continue;
|
||||||
}
|
}
|
||||||
process.stdout.write('.');
|
process.stdout.write('.');
|
||||||
await new Promise((resolve) => setTimeout(resolve, POLL_INTERVAL_MS));
|
await new Promise((resolve) => setTimeout(resolve, POLL_INTERVAL_MS));
|
||||||
@@ -664,7 +682,7 @@ async function release(options) {
|
|||||||
console.log(` push ${options.remote} ${branch}, then tags in the order above`);
|
console.log(` push ${options.remote} ${branch}, then tags in the order above`);
|
||||||
console.log(
|
console.log(
|
||||||
options.wait
|
options.wait
|
||||||
? ` wait for each version on npmjs (up to ${options.waitTimeout}s each)\n`
|
? ` wait for each version on npmjs (asks to continue after ${options.waitTimeout}s)\n`
|
||||||
: ' wait no — tags are pushed back to back\n'
|
: ' wait no — tags are pushed back to back\n'
|
||||||
);
|
);
|
||||||
|
|
||||||
@@ -751,7 +769,8 @@ async function release(options) {
|
|||||||
const landed = await waitForRelease(
|
const landed = await waitForRelease(
|
||||||
entry.pkg.manifest.name,
|
entry.pkg.manifest.name,
|
||||||
entry.version,
|
entry.version,
|
||||||
options.waitTimeout * 1000
|
options.waitTimeout * 1000,
|
||||||
|
!options.yes
|
||||||
);
|
);
|
||||||
if (!landed) {
|
if (!landed) {
|
||||||
pending.push(entry);
|
pending.push(entry);
|
||||||
@@ -772,7 +791,13 @@ async function release(options) {
|
|||||||
const blocked = pending[0];
|
const blocked = pending[0];
|
||||||
const shipped = blocked ? plan.slice(0, plan.indexOf(blocked)) : plan;
|
const shipped = blocked ? plan.slice(0, plan.indexOf(blocked)) : plan;
|
||||||
|
|
||||||
console.log(chalk.bold('\n Done\n'));
|
// The header has to know whether this worked. It was an unconditional 'Done'
|
||||||
|
// over a `shipped` list that is *empty* when the package that blocked is the
|
||||||
|
// first one — a success banner above a release that shipped nothing, with the
|
||||||
|
// diagnosis a screen further down. It read as success and was believed.
|
||||||
|
console.log(chalk.bold(blocked ? '\n Blocked\n' : '\n Done\n'));
|
||||||
|
|
||||||
|
if (blocked && shipped.length > 0) console.log(chalk.dim(' Released before the blockage:\n'));
|
||||||
for (const entry of shipped) {
|
for (const entry of shipped) {
|
||||||
console.log(` ${entry.pkg.manifest.name}@${entry.version} (${distTag(entry.version)})`);
|
console.log(` ${entry.pkg.manifest.name}@${entry.version} (${distTag(entry.version)})`);
|
||||||
console.log(chalk.dim(` npm install -g ${entry.pkg.manifest.name}@${entry.version}`));
|
console.log(chalk.dim(` npm install -g ${entry.pkg.manifest.name}@${entry.version}`));
|
||||||
@@ -817,7 +842,12 @@ program
|
|||||||
.option('--no-verify', 'Skip the lint/typecheck/test/build gate')
|
.option('--no-verify', 'Skip the lint/typecheck/test/build gate')
|
||||||
.option('--no-changelog', 'Do not prompt for release notes')
|
.option('--no-changelog', 'Do not prompt for release notes')
|
||||||
.option('--no-wait', 'Do not poll npmjs between tag pushes')
|
.option('--no-wait', 'Do not poll npmjs between tag pushes')
|
||||||
.option('--wait-timeout <seconds>', 'How long to wait for each version', Number, 1200)
|
.option(
|
||||||
|
'--wait-timeout <seconds>',
|
||||||
|
'How long to wait before asking to keep waiting',
|
||||||
|
Number,
|
||||||
|
2400
|
||||||
|
)
|
||||||
.addHelpText(
|
.addHelpText(
|
||||||
'after',
|
'after',
|
||||||
`
|
`
|
||||||
|
|||||||
Reference in New Issue
Block a user