Written 2026-09-01, before any code. The ask: a native Rust compiler,
.codex in, standard Codex IR out, in a new repo rust-codex-compiler.
Layered the way the real compiler is layered -- Rust lexer, Rust parser,
Rust desugarer, and so on. Main goal compile speed; medium-term goal
linting and bug-hunting on top. Order the corpus smallest-first. Golds
match the bare-metal compiler.
Everything below the first section is measured on this box today, not recalled.
The question was "how close did we get to that last night." Answer: the fixed-point half is already banked; the PR half is not.
upstream/master is 58b08c38 -- Update 53. Our working pin u53-rebank
is upstream/master exactly, no commits either way. The ladder carries
a u53-14of14 tag, a truth/u53 bank whose ARMS file reads agreed 14 of
14, and seed b066ceb5fe8fc9e8. So master already has a fixed point on
the ladder and it was banked on 2026-08-29. ladder_status.py reports
truths 0 recorded under this seed only because the live truth files are
regenerated artifacts; restore_truths.py puts the bank back. Re-running
tiers_run.py against restored u53 truths is a cheap confirmation, not a
discovery.
What master does not have is our work. Ten PRs are open upstream and none has merged:
| PR | branch | touches |
|---|---|---|
| 99 | prelude-parts-are-hand-written |
zig plug |
| 100 | zig-plug-real-conversions |
zig plug |
| 101 | compiler-second-pr-u53 |
compiler |
| 103 | zig-plug-drop-shadowed-arms |
zig plug |
| 105 | zig-plug-real-bitcast |
zig plug |
| 107 | foreword-real-atan |
foreword |
| 108 | foreword-cordic-accuracy |
foreword |
| 111 | plugs-u53-safari-batch |
plugs |
| 112 | wasm-plug-selfhost-batch |
wasm plug |
| 114 | lexer-eol-text-literal |
core compiler |
The all-outbound-u53 branch exists and is a child of 58b08c38, but it
integrates seven of the ten -- 99, 100, 101, 103, 105, 107 and a plug
run.ps1 fix. 108, 111, 112 and 114 are missing from it. That branch is
last week's integration, not this week's.
We cannot make upstream master carry our fixes. What we can do is build
the branch that predicts it, and re-pin when Damian lands them. Call it
master-plus-outbound. Two of the ten change what the front end produces,
which is why this is not optional:
So: golds come from master-plus-outbound, not from master.
The stale-bank warning. native/codexir was built on 2026-08-26 from
pin a961dcb6, and corpus/*.ir is a bank from around then. They no longer
agree with each other in every case. db-full-test differs from its banked
IR today: the bank carries two (forall ...) wrappers this build does not
emit, and this build pulls in a Unicode Code Point Boundary chapter the
bank never saw -- that second one is the case-insensitive quire lookup fix
from 2026-08-30 recovering a chapter that had been silently dropped.
arithmetic, web-chain and brotli-interop match the bank byte for byte.
Conclusion: regenerate the whole gold set. Do not trust corpus/*.ir.
The 14 rungs are, in ast/oracle_lib.sh order:
lex parse desugar scope check lower
ir_to_codex ir_to_codex_roundtrip lir_to_x86 ir_to_wire
ir_to_x86_on_fib ir_to_x86_on_cce
passes_to_x86_on_mid passes_to_x86_on_arith
The first six are exactly the Rust layers. Rungs 7-14 are back end -- x86, LIR, the wire -- and the Rust project stops at IR, so it does not climb them.
And the truths are already the right shape for a port. u53-lex.truth is a
token dump, 172 KB, 5,339 tokens, one line each:
TypeIdentifier 0+7 L1C1 |Chapter|
Colon 7+1 L1C8 |:|
TypeIdentifier 9+5 L1C10 |Lexer|
Newline 14+1 L1C15
Offset, length, line, column, text. That is a finished acceptance test for a Rust lexer, banked, with provenance, generated by the bare-metal compiler. Nothing needs designing.
u53-check.truth is better still. It is 844 bytes over Fib -- three
definitions -- and it ends:
substitutions 8
next-id 8
expr-types 11
next-id is the unification counter, exposed. Which brings us to the
one decision that shapes the entire project.
Look at a real IR line from corpus/arithmetic.ir:
(apply (name "print-line-uni" (fn text nothing
(row (labels (label "Console.Write" "")) "" 467))) ...)
That 467 is row.tail-id -- a fresh unification variable allocated by the
type checker, printed into the IR text. It is not incidental. Across the 580
banked programs:
(row ...) sites, and 520 of 580 files carry a nonzero
tail-id. Values run into the thousands; 7027 appears 100 times.(tvar N) occurrences across 74 files.Byte-identical IR therefore requires reproducing the exact order in which the Cobblestone type checker allocates fresh variables, across the whole program. Not the same types. The same counter sequence. That is a faithful port of the inference traversal, and it is in direct tension with "we want ours to be better" -- any improvement to the inference walk moves every number downstream of it.
My recommendation, and I think this needs your ruling before line one:
The standing gate is canonical equality; byte-identity is a ratchet, not a gate.
Canonicalize both sides by renumbering tvar and row tail-ids in
first-occurrence order, then compare. An alpha-equivalent IR is a correct
IR; a differently-numbered one is not a defect and should not be able to
fail a build. Separately, count how many programs come out byte-identical
and ratchet that number upward -- never down. That keeps the honest signal
(a canonical mismatch is a real defect, fail loud) without making month one
impossible, and it keeps the door open to a better checker.
The cost of getting this wrong in the other direction: we spend the project reverse-engineering an allocation order, and the first algorithmic improvement we make turns the whole corpus red for no reason.
next-id in the check truth is the instrument that tells us how far apart
the two allocators are, program by program, cheaply. Use it.
Measured today with cite_resolve.py over codex/test/**:
foreword-all-compile).Two things follow that change the smallest-first plan.
There is no hello-world. The floor is 3.7 KB because every unit gets
Foreword ListUtils and Foreword Tuple prepended -- the desugarer writes
calls to map-list and MkTupN that no author would know to cite. The
"smallest program" is 3.7 KB of library plus five lines.
Refinement: for rungs 1-3, work on a synthetic micro-corpus that skips
cite resolution entirely. The ladder already does this -- the check and
lower rungs run on Fib, 85 tokens, three definitions. Ten-line inputs
for the lexer and parser; the 3.7 KB floor only starts mattering at scope.
The smallest real programs are error tests. The bottom of the size list
is unterminated-text, arith-on-text, if-no-else, let-no-in,
type-mismatch, unknown-name, duplicate-ctor. Their gold is a
diagnostic, not IR.
Refinement: partition before sorting. Corpus A = programs whose gold is clean IR; corpus B = programs whose gold is a CDX diagnostic. Climb A smallest-first. B is not throwaway -- B is where the linting goal lives, and matching the bare-metal compiler's diagnostic codes and positions is the foundation the linter sits on. But it is the second corpus, not the first seven entries of the first one.
native/codexir runs here today, no VM, and it is deterministic (two runs
byte-identical). Measured:
| program | resolved src | IR out | time | throughput |
|---|---|---|---|---|
arithmetic |
5.8 KB | 7.2 KB | 0.051 s | ~113 KB/s |
web-chain |
227 KB | 783 KB | 0.68 s | 324 KB/s |
brotli-interop |
430 KB | 857 KB | 0.74 s | 572 KB/s |
db-full-test |
281 KB | 1.13 MB | 2.57 s | 107 KB/s |
Call it ~150 KB/s typical. The full 225 MB corpus is therefore about
25 minutes through codexir -- which is also the cost of regenerating
the entire gold set, once, and that is cheap enough to do on every re-pin.
The target I would write down: the Rust front end emits IR for all 1,700 programs in under 60 seconds single-threaded, and under 10 seconds across the box's cores. That is roughly 100x, and it is a modest ask for Rust against a compiler that is bootstrapped through its own IR. Writing the number down now is what makes it a goal rather than a vibe.
Phase 0 -- the pin (no Rust yet).
master-plus-outbound = 58b08c38 + all ten PR branches. Seven
are already integrated in all-outbound-u53; add 108, 111, 112, 114.native_build.sh against it. This is a QEMU job, ~13 minutes, and it is
the one step in the whole plan that needs the appliance.restore_truths.py + tiers_run.py -- expect 14/14. This is the
sanity check you asked for, and it is now a check on the integration
branch, not on master, which is the more useful question.PROVENANCE file into the Rust repo recording pin, seed, and
natives stamp. BOX 9: read the provenance of the comparand. Last night
cost a red run for exactly this.Phase 1 -- golds. One script, regen_golds.py: resolve, run codexir,
capture stderr only (stdout carries CX-DECK noise), write
golds/<name>.ir plus a manifest of sizes and hashes. ~25 min. Never
hand-edit a gold; the script is the fix.
Phase 2 -- Rust. rustup, not apt -- apt here has 1.75, too old. Then
the repo, then the rungs in order: lexer against u53-lex.truth, parser
against u53-parse.truth, desugarer, scope, check, lower.
1. The tvar-numbering decision (section 3). Highest leverage thing on this page. Needs your call.
2. Design the parser lossless from day one. Keep trivia -- whitespace, comments, exact spans -- in a concrete syntax tree, and lower to the AST from it. The medium-term linting goal wants a CST and fix-its; retrofitting one onto an AST-only parser later is a rewrite, not a refactor. The runtime cost is small and the speed goal survives it. This is the one place I would not copy the Cobblestone design, which throws the trivia away.
3. The desugarer is where the semantics hide. for x in xs -> ...
becomes a map-list call; a tuple literal becomes MkTup<N>. Those are
invisible in the source and load-bearing in the IR. It is the layer most
likely to have behaviour that is in the code and nowhere else, and the
layer where a wrong guess produces plausible IR that is subtly wrong.
4. Scale honesty. The Cobblestone compiler is 57,364 lines of Codex
across 64 chapters: Types/ 8,518, IR/ 8,184, Syntax/ 5,060, Ast/
2,112, Semantics/ 1,101, Emit/ 28,061. The front-end slice we need to be
equivalent to is roughly 20-25 K lines. This is a real project. I would not
promise IR parity in a week.
5. The pin will move under us. "Taken mostly verbatim" is mostly. Any
edit Damian makes shifts golds. Mitigation is already in the plan -- 25
minute regeneration and a --diff-golds report that says which programs
moved and by how much -- but it should be a habit from day one, not a
reaction.
6. This competes with the ladder. Ten PRs are open and the box is one machine. Worth being deliberate about which of the two gets it.
Correction, same day: this risk originally said the top ladder item was
ours -- that ast/emit_harness.py still generated the broken -halted
literal. It does not. Ladder fddbf71 fixed the generator last night and
verified it end to end under a PR-114 compiler; all four repo copies carry
the fix and both fixed points were rebuilt on it (codexzig-safari 316f9ce,
codex-wasm-transpiler 8c8041a). I had taken the claim from a memory index
without checking it against the commit. What IS still open is smaller:
codex-zig-transpiler's generated/ is from 08-30 and its harness fix is
98f5959 from 09-01, so that repo alone is one rebuild behind -- about ten
minutes of codexzig_build.sh.
7. Do not let the Rust compiler become a second oracle by accident. The bare-metal compiler is the oracle. When Rust disagrees with it, the default assumption is that Rust is wrong. The moment we start filing findings because Rust says so, we need the same due-diligence chain we already have -- and a disagreement that turns out to be theirs is the best possible outcome, but it has to be earned the same way.
master-plus-outbound rather than plain master.emit_harness
item.