feat(custom-model): live countdown on the loading banner; timeout is now an error

The loading banner now shows a live countdown against its own timeout
(updated every poll, so every second by default) instead of a static
"this can take a while" — e.g. "Loading qwen3.8-27b (16.4 GB, typically
~1-3 min) on llama-swap - 47s remaining".

If the countdown reaches zero and the model still isn't ready, this is now
treated as a real failure rather than a "keep waiting" shrug:
- The banner turns into a sticky error (_showCenterStatus gains a `type`
  option - 'error' drops the spinner and adds a close button, since nothing
  is "in progress" anymore and a sticky message needs a way to dismiss it),
  naming the llama-swap server's own logs as where to look for detail.
- The session that load was for is closed automatically (closeSession) -
  requested explicitly: a console left open and pointed at a model that
  never finished loading is worse than no console at all. Both apply paths
  now thread the new session's id through to _watchLlamaSwapLoading for
  this (new required 3rd parameter, after endpointId/modelId).

_watchLlamaSwapGeneration's existing stale-call guard extends naturally to
this: a superseded call's own eventual timeout recognises it no longer owns
the banner and neither shows the error nor closes a session that may by
then belong to a different, newer launch.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RqZeHrRS6DYcGcGX2p9EwG
This commit is contained in:
Devvyn
2026-09-16 20:22:02 +08:00
co-authored by Claude Sonnet 5
parent 55dae31530
commit 7bbe408e44
7 changed files with 181 additions and 64 deletions
+44 -16
View File
@@ -765,7 +765,7 @@ Object.assign(CodemanApp.prototype, {
const result = this._lastCustomModelLaunchResult;
this._lastCustomModelLaunchResult = undefined;
if (result?.modelSwapInProgress) {
void this._watchLlamaSwapLoading(endpointId, modelId);
void this._watchLlamaSwapLoading(endpointId, modelId, result.sessionId);
}
},
@@ -906,7 +906,7 @@ Object.assign(CodemanApp.prototype, {
// Hand off to its own sticky toast rather than stacking a second one on top.
if (payload?.modelSwapInProgress) {
switchingToast?.dismiss();
void this._watchLlamaSwapLoading(endpointId, modelId);
void this._watchLlamaSwapLoading(endpointId, modelId, sessionId);
return;
}
@@ -967,29 +967,43 @@ Object.assign(CodemanApp.prototype, {
return this._MODEL_LOAD_TIME_MATRIX.find((bracket) => sizeGB <= bracket.maxGB) ?? null;
},
/** `ms` -> `"1m 08s remaining"` / `"8s remaining"`, for the loading banner's live countdown. */
_formatRemaining(ms) {
const totalSec = Math.max(0, Math.ceil(ms / 1000));
const mins = Math.floor(totalSec / 60);
const secs = totalSec % 60;
return mins > 0 ? `${mins}m ${String(secs).padStart(2, '0')}s remaining` : `${secs}s remaining`;
},
/**
* Polls llama-swap's own `/running` (via the read-only running-status route) until
* `modelId` reports `state: 'ready'`, showing a sticky banner the whole time so a slow
* unload/reload (measured well over a minute for a large model) reads as "loading",
* never as silence or a wrong answer from whatever was loaded before. Checks immediately
* (a fast load, or a re-apply onto an already-ready model, shouldn't wait a full interval
* to say so), then every `pollIntervalMs`. Bounded at `maxWaitMs` — defaults to a rough,
* size-scaled estimate (`_estimateModelLoad`) when the model's discovered size is known,
* falling back to a flat 5 minutes when it isn't; still not ready by then gets a toast
* saying so rather than polling forever.
* `modelId` reports `state: 'ready'`, showing a sticky banner with a live countdown the
* whole time so a slow unload/reload (measured well over a minute for a large model)
* reads as "loading, N seconds left", never as silence or a wrong answer from whatever
* was loaded before. Checks immediately (a fast load, or a re-apply onto an
* already-ready model, shouldn't wait a full interval to say so), then every
* `pollIntervalMs`. Bounded at `maxWaitMs` — defaults to a rough, size-scaled estimate
* (`_estimateModelLoad`) when the model's discovered size is known, falling back to a
* flat 5 minutes when it isn't.
*
* If the countdown reaches zero with the model still not ready, this is a real failure,
* not a "keep waiting" — the banner turns into a sticky error naming the llama-swap
* server's own logs as where to look, and `sessionId` (the session this was launched
* for) is closed automatically: a console left open and pointed at a model that never
* finished loading is worse than no console at all.
*
* `_watchLlamaSwapGeneration` guards against two overlapping calls (a second launch
* started before the first one's loop finished) clobbering each other's banner:
* `_showCenterStatus` reuses one shared DOM node, so an older loop's `dismiss()`/message
* update firing after a newer one has already taken over the banner would otherwise hide
* or overwrite the WRONG one. Each call claims the counter as its own "generation" and
* checks it still owns it before touching the banner.
* or overwrite the WRONG one, or close the WRONG session. Each call claims the counter
* as its own "generation" and checks it still owns it before touching either.
*
* `pollIntervalMs`/`maxWaitMs` exist to let a test drive this in milliseconds instead of
* minutes — real callers never pass `maxWaitMs`, which is what keeps the size-scaled
* default live here rather than only in a test fixture.
*/
async _watchLlamaSwapLoading(endpointId, modelId, pollIntervalMs = 1000, maxWaitMs) {
async _watchLlamaSwapLoading(endpointId, modelId, sessionId, pollIntervalMs = 1000, maxWaitMs) {
const generation = (this._watchLlamaSwapGeneration = (this._watchLlamaSwapGeneration || 0) + 1);
const isCurrent = () => this._watchLlamaSwapGeneration === generation;
const sizeGB = await this._lookupModelSizeGB(endpointId, modelId);
@@ -999,10 +1013,11 @@ Object.assign(CodemanApp.prototype, {
const sizeSuffix = sizeGB
? ` (${sizeGB.toFixed(1)} GB${estimate ? `, typically ${estimate.label}` : ''})`
: '';
const baseMessage = `Loading ${modelId}${sizeSuffix} on ${endpointId} —`;
// Prominent and screen-centred, not a corner toast — a real llama-swap model load can
// sit on screen for well over a minute, easy to mistake for nothing happening there.
const toast = this._showCenterStatus(`Loading ${modelId}${sizeSuffix} on ${endpointId}… this can take a while`);
const deadline = Date.now() + effectiveMaxWaitMs;
const toast = this._showCenterStatus(`${baseMessage} ${this._formatRemaining(deadline - Date.now())}`);
while (Date.now() < deadline) {
const status = await this._apiJson(`/api/model-endpoints/${encodeURIComponent(endpointId)}/running-status`);
if (!isCurrent()) return; // a newer launch took over the banner — this loop is done
@@ -1018,11 +1033,24 @@ Object.assign(CodemanApp.prototype, {
this.showToast(`${modelId} is ready`, 'success', { duration: 2500 });
return;
}
if (!isCurrent()) return;
toast?.setMessage(`${baseMessage} ${this._formatRemaining(deadline - Date.now())}`);
await new Promise((resolve) => setTimeout(resolve, pollIntervalMs));
}
if (!isCurrent()) return;
toast?.dismiss();
this.showToast(`Still waiting for ${modelId} to finish loading on ${endpointId} — check the llama-swap server`, 'warning');
this._showCenterStatus(
`${modelId} did not finish loading on ${endpointId} within the expected time. ` +
`Check the llama-swap server logs for details.` +
(sessionId ? ' The session has been closed.' : ''),
{ type: 'error' }
);
if (sessionId) {
try {
await this.closeSession(sessionId);
} catch {
// closeSession already reports its own failure via toast — nothing more to do here
}
}
},
/**