mirror of
https://github.com/Ark0N/Codeman.git
synced 2026-09-30 12:39:42 +02:00
test(qr-auth): replace flaky max-deviation bias check with chi-square
The short-code distribution test asserted that no base62 character deviated more than 15% from its expected count. That statistic is the maximum over 62 correlated near-normal cells, so its tail is fat: at n=36000 the per-cell relative SD is ~4.1%, which puts the 15% bound at |z| ~ 3.65, and taken as a max over 62 cells it fires on a perfectly uniform generator about 1.6% of the time. Measured directly: 48 spurious failures in 3000 simulated runs. It had been rerun-to-green repeatedly and most recently red-herringed a PR review. Chi-square is the correct test for "is this multinomial uniform", and unlike 0.15 its threshold is derivable. df=61, Wilson-Hilferty puts the p=1e-6 critical value at ~129, so the bound is 130. Power is unchanged. Removing rejection sampling from generateShortCode reintroduces modulo bias (256 % 62 = 8, so eight characters draw five chances per 256 instead of four) and was verified against the real code in an isolated worktree: chi-square 243.06 against the 130 limit. The threshold sits in a wide empty gap, 3000 clean runs peaked at 104 while 200 biased runs bottomed out at 174.5. Also iterate the alphabet explicitly rather than the observed keys, so a character that never appears counts as zero instead of being skipped. Verified: 30/30 consecutive runs of the real test pass. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
+31
-9
@@ -250,6 +250,9 @@ describe('QR Token Manager (unit)', () => {
|
|||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
|
/** Must match the alphabet in `generateShortCode` (tunnel-manager.ts). */
|
||||||
|
const BASE62_ALPHABET = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789';
|
||||||
|
|
||||||
describe('Short code distribution (bias check)', () => {
|
describe('Short code distribution (bias check)', () => {
|
||||||
it('should produce roughly uniform character distribution', () => {
|
it('should produce roughly uniform character distribution', () => {
|
||||||
// Generate 6000 codes (36000 chars) and check distribution
|
// Generate 6000 codes (36000 chars) and check distribution
|
||||||
@@ -265,17 +268,36 @@ describe('Short code distribution (bias check)', () => {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// Expected count per char: 36000 / 62 ≈ 580.6
|
// Chi-square goodness-of-fit against a uniform base62 alphabet.
|
||||||
const expected = 36000 / 62;
|
//
|
||||||
let maxDeviation = 0;
|
// This deliberately does NOT assert on the max per-character deviation.
|
||||||
for (const [, count] of charCounts) {
|
// That statistic is the maximum of 62 correlated near-normal cells, so its
|
||||||
const deviation = Math.abs(count - expected) / expected;
|
// tail is fat: with n=36000 the per-cell relative SD is ~4.1%, which puts a
|
||||||
maxDeviation = Math.max(maxDeviation, deviation);
|
// 15% bound at |z| ~ 3.65 and, taken as a max over 62 cells, fails on a
|
||||||
|
// perfectly uniform generator about 1.6% of the time. Measured over 3000
|
||||||
|
// simulated runs: 48 spurious failures. That is the flake.
|
||||||
|
//
|
||||||
|
// Chi-square is the right tool for "is this multinomial uniform", and its
|
||||||
|
// threshold is derivable rather than eyeballed. df = 62 - 1 = 61, so under
|
||||||
|
// the null E[X²] = 61 and SD = sqrt(2*61) ~ 11.05; the Wilson-Hilferty
|
||||||
|
// approximation puts the p = 1e-6 critical value at ~129. Rounding to 130
|
||||||
|
// gives a false-positive rate around one run in a million.
|
||||||
|
//
|
||||||
|
// Power is unaffected. Dropping rejection sampling reintroduces modulo bias
|
||||||
|
// (256 % 62 = 8, so the first 8 characters draw 5 chances per 256 instead
|
||||||
|
// of 4, ~25% overrepresented), which scores X² ~ 237. Simulated: 3000 clean
|
||||||
|
// runs peaked at 104, while 200 biased runs bottomed out at 174.5, so the
|
||||||
|
// threshold sits in a wide empty gap between the two.
|
||||||
|
const alphabetSize = 62;
|
||||||
|
const expected = 36000 / alphabetSize;
|
||||||
|
let chiSquare = 0;
|
||||||
|
for (let i = 0; i < alphabetSize; i++) {
|
||||||
|
const count = charCounts.get(BASE62_ALPHABET[i]) ?? 0;
|
||||||
|
chiSquare += (count - expected) ** 2 / expected;
|
||||||
}
|
}
|
||||||
|
|
||||||
// With rejection sampling, deviation should be < 15% (generous)
|
expect(charCounts.size).toBe(alphabetSize);
|
||||||
// Without rejection sampling (modulo bias), first 6 chars would be ~25% overrepresented
|
expect(chiSquare).toBeLessThan(130);
|
||||||
expect(maxDeviation).toBeLessThan(0.15);
|
|
||||||
|
|
||||||
tm.stopTokenRotation();
|
tm.stopTokenRotation();
|
||||||
});
|
});
|
||||||
|
|||||||
Reference in New Issue
Block a user