KERNEL FORENSICS

The wall you could weigh

Chapter 2 ended with a restart loop; chapter 3 ended with real interrupt fixes and a wall that had moved downstream to meet them. By mid-July the failure had changed character entirely: the installer running inside mddosem no longer failed loudly. It simply stopped. Every capture run ended the same way — a black 640×480 frame. Not approximately black: the automated screenshot rig wrote the same PNG file over and over, exactly 5,757 bytes, byte-for-byte identical, across four days of captures. Every progress-*.png, every sample-*.png; days later, even the first attempt to launch FreeCell on the freshly installed system produced the identical 5,757 bytes. The wall was the only exhibit in the archive you could identify by weight.

A black screen is an absence of information; a byte-identical black screen is a measurement. It said, precisely, that nothing on the display side had changed at all — and that whatever the machine was doing, it was doing it in the dark. Getting at it would take instruments, and the instruments escalated.

The escalation ladder

An emulator is the best debugger there is, because it is also the hardware. mddosem can pause its guest between any two instructions and read anything — registers, memory, device state — without the guest's consent or knowledge. That power is exposed through a control pipe (--ctl-pipe): a named pipe on the host accepting commands — pause, step, read registers, read memory. The first instruments were hand-rolled Python scripts speaking that pipe — dbg.py, dbgstack.py, dbgbytes.py — a debugger assembled in an afternoon, one command at a time. Here it is diagnosing an earlier freeze in NTLDR, Windows' boot loader, live:

That is a diagnosis no screenshot could make: the instruction pointer pinned, interrupts off, single-step refusing to advance — a CPU re-entering the same fault forever, distinguishable from an ordinary busy-loop only when you can hold the machine still and poke it. The pipe scaled up from there. When runs started dying in blue screens, the harness polled the kernel's own confession: NT records every crash's stop code and parameters in a kernel variable, KiBugCheckData, before it paints the screen, and polling it live caught a STOP 0x1E within about four seconds of it happening. When live inspection was not enough, a RAMDUMP command froze the guest's entire physical memory to disk, and offline scanners walked the registry's hive cells in the captured images. And when all of that still watched from outside the kernel's own machinery, there was one instrument left — the one Microsoft built.

A kernel debugger from spare parts

Every Windows NT kernel ships with a debug stub. Boot it with /debug /debugport=com1 /baudrate=19200 in boot.ini and KDCOM, the kernel-debug serial transport, speaks a packet protocol out of COM1, letting a second machine stop the kernel, walk its structures, and name its threads. Inside an emulator the "serial port" is the emulated 16550 UART and the "second machine" is the host, so the transport chain became: WinDbg ⇄ a named pipe ⇄ a relay ⇄ TCP ⇄ the emulated COM1.

It was proven link by link, from the guest outward. The first witness was comtx.com — a 79-byte DOS program, written in assembler for the occasion, which programs the UART at port 0x3F8 and transmits KDTEST-COM1- forever. The host capture received 57,960 bytes of the clean repeating string, and under load the transport measured effectively lossless: 4,139 clean records, 14 stray bytes — 0.024% — and exactly one resync, at the connection join.

Then a human-shaped blocker. boot.ini lives on an NTFS volume, and no headless tool on the bench could edit it: mddskimg, the fleet's own disk-image inspector, falsely reported "NTFS directory loop detected" and refused. Arthur fixed mddskimg's NTFS reader the same day (v0.57.5 → v0.57.6); the patch itself went in through an ntfs-3g loop mount, and the /debug-patched disk went back into the emulator. What came out was quietly historic for this project: Windows 2000's own KDCOM emitted 161 valid KD_STATE_CHANGE64 packets through the emulated UART — correct leaders, correct types, payloads carrying genuine ntoskrnl code:

Later that night the answering side arrived too: kd.exe, the command-line debugger, was dug out of the WinDbg Preview Store package and completed the full KDCOM handshake, scripted. And then the anticlimax. Symbols are the map from raw addresses to names — and Microsoft's symbol server no longer carries Windows 2000. The SP4 CD ships none. Without symbols, !process and !thread are blind: the debugger connected perfectly to a kernel it could no longer name. So the team kept the transport as a proof, and pivoted to something better suited to an emulator anyway — a debugger that needs no symbols at all.

The dump and the census

At 23:12 on 14 July the hung machine's entire physical memory — exactly 128 MiB, 134,217,728 bytes — was preserved as hang.bin, its SHA-256 recorded beside it. Everything an operating system knows, it knows in RAM: the process list, every thread, every wait queue, every object name. A memory dump is the whole crime scene. What you need is a map of the rooms.

The map came as a tool: w2kcensus, a purpose-built forensic walker checked into the emulator's own source tree — initially 1,172 lines of paranoid Rust. It refuses to run unless the kernel in front of it fingerprints as exactly the SP4 build — ntoskrnl 5.00.2195.6611, PE timestamp 0x3EE6C002. It finds the process list the way a debugger would — resolve PsInitialSystemProcess from the kernel's export table, then walk the EPROCESS list — but it validates every hard-coded structure offset by reciprocity: doubly-linked lists must close (follow Flink forward and Blink back, and you must land where you began); a thread's wait blocks must point back at their thread; pool allocations must carry the right tags. It walks 4 KiB and 4 MiB pages, reads object names only through the kernel's own OBJECT_HEADER_NAME_INFO records, and works both live over the control pipe and offline over the dump — the offline run reproduced the live findings exactly.

The paranoia paid immediately, because the published literature was wrong. The standard reference for NT structure offsets — the Geoff Chappell "3.51–5.1" tables — places KTHREAD.WaitBlockList at 0x5C. In this kernel it is at 0x58; that four-byte error "was why earlier wait-object reads were garbage". ThreadListEntry likewise: 0x1A4, not the published 0x110.

The census returned a result of eerie clarity: 8 processes, 71 non-idle threads, and every single one of them Waiting. Zero ready. Zero running. The CPU parked in the kernel's idle loop at 0x800695A2, executing HLT, with only the timer interrupt firing. And setup.exe — PID 352, the process this whole campaign existed to shepherd — had exactly three threads, each blocked on its own anonymous, unsignalled synchronisation event, their kernel stacks swapped out to disk, something NT does only to a thread it expects to sleep for a long time. The verdict came from the live census, recorded at 06:38 the next morning, and wrote itself: "a genuine whole-system lost-wakeup deadlock: everyone blocked, nothing to wake them." Later that day w2kcensus reproduced it exactly, offline, over hang.bin (address space CR3 0x00030000).

The retractions

Forensic instruments lie, and the record of this week is unusually honest about catching them. Three corrections, all self-inflicted and all found by the team's own cross-checks:

• The named event that wasn't. The first census reported one of setup's three events as $$$SetupBackground — a name, at last, something to search for. But the string exists in exactly one binary on the disk, syssetup.dll, where it is a window-class name (a WNDCLASSW.lpszClassName), not an event name. In the dump it sits in an Atom-tagged pool allocation; the true event is the next allocation along, tagged Even, whose object header says NameInfoOffset=0 — structurally anonymous. The census had walked off the end of one allocation and christened the neighbour.
• The ninth process. An early walk reported nine processes; the ninth was the process list's own head — the sentinel node at 0x8046E460 — read as if it were an EPROCESS. Lists that point to themselves will happily impersonate their members.
• The field at KTHREAD+0x44 — pinned down as ApcState.Process, the thread's owning-process backpointer, and thereafter used as one more reciprocity check rather than trusted on faith.

And the instrument's heaviest confession: READ_MEM, the live memory-read command, was found to be disturbing the evidence — setting Accessed bits in the guest's page tables, populating its TLB, perturbing pending page-fault state. It was rebuilt as a genuinely side-effect-free supervisor page walk, and the pause command was made atomic so the watcher could never race a human's pause. One rejected snapshot even caught Windows mid-update of a wait block — a pointer transiently aimed into a stack page — exactly the kind of half-written state you want a tool to reject rather than report. The same discipline killed hypotheses, too: the week's leading rival theory — that the emulator was coalescing two interrupt completions into one — was refuted not by argument but by a counter, a log-only probe recording 0 coalescing events across 291 FLUSH CACHE commands and 15,418 disk write-commits. Measurement over belief, every time.

The observer effect

Live censuses came with a cost nobody had priced in. Capture 06 sampled the machine every second — and the sampling consumed roughly 900 of the run's 3,000 wall-seconds. Worse, the emulator's adaptive scheduler, which tunes guest speed against real time, read the watcher's stalls as its own slowness and dragged itself from 65,000 cycles per millisecond down to the low hundreds. The result was 1,602 flawless censuses of a machine that never hit the wall — because the machine under observation was no longer the machine that failed. Capture 07 relaxed the cadence to five seconds and logged errors only: the guest ran true, and the watcher's time bound expired seconds before the terminal transition it existed to catch. The very next census found the wall fully formed — kernel stacks non-resident, cycles_per_ms=65000, halted=true. Two runs, one lesson Heisenberg could have countersigned.

The consulting diagnostician

Threaded through the week was a third kind of instrument: an outside mind. Fable 5 — a consulting model engaged for finite, expensive reasoning sessions — was handed the code and the evidence, and its findings were treated exactly as the census's were: every load-bearing claim independently re-verified against source before being written down. Its root-cause analysis ranked the field (a storage-path wedge at ~55%, a genuine Windows race made probable by emulator timing at ~20%, a lost timer wakeup at ~10%) and flagged the strangest numbers of the night: the DSS cryptography providers loading pathologically slowly — dssbase in 28.7 seconds, dssenh in 28.5, against three to four for their RSA siblings. Its advice on that ambiguity became a house rule: "Don't bet — pause inside the 28 s window and read the stack."

The machine that was not asleep

Which leaves the paradox the census could describe but never resolve. All the instruments agreed: 71 threads, all waiting; zero runnable; the CPU halted in the idle loop. A dead calm. And yet — during one 150-second stretch of that "zero-runnable" state in a later run (capture 09), the emulator counted 2,789 protected-mode stack transitions: about 18.5 every second. A stack transition is what happens when an interrupt or exception crosses a privilege boundary — hardware still delivering, the kernel still fielding its timer ticks, looking around, finding no one worth waking, and lying back down. The machine was not asleep. It was awake, attentive, and waiting, with the composure of something that knows exactly what it is waiting for.

The census could count the sleepers to the last thread. What it could not say was what would wake them — because the answer was not in the dump at all. It was sitting in plain sight, in the dark, one keystroke away.

Sources for this chapter — the KD-over-serial bring-up journal (2026.07.14), the thread-census journal and the endgame master journal (2026.07.15), the Fable 5 root-cause consult HLD (2026.07.14), the bug ledger MDD-BUG-KILN-00004, and the preserved run archive (hang.bin, the capture-04→29 black frames, FINDINGS.md). The census tool is src/bin/w2kcensus.rs in mddosem. Full evidence table: how we know.