Status: staged for review
Date: 2026-06-17
Source repo: G:\repos\GPUPerft
Source branch: codex/exact-export-exe
Primary question: what board representation and compression path can persist generated Frontier positions at the highest physical rows per second?
Why This Experiment Was Run
GPUPerft can generate positions far faster than the current export path can persist them. The exporter therefore has to be treated as a pipeline: generation, representation, sorting, compression, transfer, and storage all have to be sized against the narrowest pipe.
The governing relationship is:
persisted_positions_per_second =
storage_bandwidth_bytes_per_second * 8 / bits_per_position
That makes two things matter at the same time:
- Rows per second: the compressor and writer must not stall the generator.
- Bits per position: lower density means the same disks can persist more positions per second.
The experiment was deliberately isolated from full d9 export at first. The point was to avoid guessing inside a complicated end-to-end run and instead measure the codec and encoding choices directly.
What Was Tested
The input was representative d6 Frontier export data from the current GPUPerft/NybbleBoard work:
- NB32 board rows
- NBM40 board plus multiplicity rows
- NB-state48 rows containing board, state, and multiplicity
The matrix varied:
- board representation
- piece-code mapping
- whether castling and en-passant state were stored as sidecar state bytes or encoded directly as piece/symbol values
- sorted vs unsorted records
- row, byteplane, XOR-row, and XOR-byteplane transforms
- chunk sizes from 256 KiB through 16 MiB
- nvCOMP ANS, Cascaded, and LZ4
- Cascaded as both a generic transformed buffer and as semantic typed columns
The Cascaded distinction matters. Cascaded is designed for columns of typed integer-like data. Throwing whole rows at it is not a fair test. A follow-up harness split the records into semantic lanes:
- one
ucharlane per board byte - one decoded
ucharlane per square symbol ucharorushortstate lanesulonglongmultiplicity lanesulonglongoccupancy lanes- one
ucharlane per occupied-piece slot - four
ulonglongsymbol-bitplane lanes
Best Results
Best Sorted ANS Throughput
| Rank | Encoding and codec | Persisted rows/sec | Bits/position |
|---|---|---|---|
| 1 | NB-state41, frequency-coded pieces, 1-byte state sidecar, sorted, ANS, XOR-byteplane, 4 MiB chunks | 2.460 B | 26.126 |
| 2 | NB-state40, piece-pair symbols, castling/en-passant encoded as symbols, sorted, ANS, XOR-byteplane, 4 MiB chunks | 2.443 B | 25.889 |
| 3 | NB-state41, piece-pair symbols with castling as symbols and en-passant as byte sidecar, sorted, ANS, XOR-byteplane, 4 MiB chunks | 2.427 B | 25.895 |
| 4 | NB-state40, frequency-coded pieces with castling/en-passant encoded as symbols, sorted, ANS, XOR-byteplane, 4 MiB chunks | 2.423 B | 25.937 |
| 5 | NB-state42, piece-pair symbols, 2-byte state sidecar, sorted, ANS, XOR-byteplane, 4 MiB chunks | 2.420 B | 26.088 |
Best Density
| Rank | Encoding and codec | Persisted rows/sec | Bits/position |
|---|---|---|---|
| 1 | NB-state40, piece-pair symbols, castling/en-passant encoded as symbols, sorted, ANS, XOR-byteplane, 256 KiB chunks | 532.269 M | 24.328 |
| 2 | NB-state40, frequency-coded pieces with castling/en-passant encoded as symbols, sorted, ANS, XOR-byteplane, 256 KiB chunks | 525.103 M | 24.404 |
| 3 | NB-state41, piece-pair symbols with castling as symbols and en-passant as byte sidecar, sorted, ANS, XOR-byteplane, 256 KiB chunks | 488.049 M | 24.404 |
Cascaded Fairness Check
| Cascaded shape | Optimistic parallel-lane rows/sec | Conservative serial-lane rows/sec | Bits/position |
|---|---|---|---|
| NB42 frequency-coded pieces, 2-byte state sidecar, semantic lanes, 256 KiB chunks | 2.542 B | 160.223 M | 56.587 |
| Decoded square-symbol lanes, frequency-coded symbols, 256 KiB chunks | 1.371 B | 34.440 M | 55.894 |
| Decoded square-symbol lanes, frequency-coded symbols, 1 MiB chunks | 471.614 M | 11.539 M | 55.702 |
| Occupancy plus split occupied-piece slot lanes, frequency-coded symbols, 256 KiB chunks | 780.092 M | 88.014 M | 116.948 |
| Generic transformed-buffer Cascaded best | 684.057 M | n/a | 89.362 |
What The Data Says
The current isolated winner is:
sorted NB-state rows
piece-pair symbols
castling and en-passant encoded as symbols
ANS compression
XOR-byteplane transform
4 MiB chunks for throughput
256 KiB chunks for density
The absolute fastest isolated row rate was the frequency-coded, 1-byte-state-sidecar ANS shape at 2.460B rows/sec. The more attractive implementation candidate is the piece-pair, state-as-symbol shape at 2.443B rows/sec and 25.889 bits/position, because it is nearly as fast and denser.
The state-as-symbol result is important. It supports the same general idea that made NybbleBoard and CCCF effective: encode chess state into the board-symbol stream when that makes the data more regular and compressible.
What Failed
Generic Cascaded failed. Treating transformed row buffers as one Cascaded payload produced poor density and did not compete with ANS.
Fair lane-aligned Cascaded still failed for this data. Cascaded was then given typed semantic columns, including decoded square-symbol lanes. That made the test fairer, but the best lane-aligned result still needed about 56 bits/position, roughly twice the sorted ANS result.
LZ4 failed. It was neither fast enough nor dense enough to compete.
Unsorted routes failed. Sorting materially improved both compression density and persisted rows/sec.
Chart
The chart shows the practical frontier: sorted ANS/XOR-byteplane sits in the fast-and-dense corner. Cascaded can approach the same row/sec only under optimistic parallel lane assumptions, and it does so at more than double the bits per position.
What This Does Not Prove Yet
This was an isolated codec and representation lab. It does not yet prove the full exporter can sustain these rates end to end.
The next proof must be a real GPUPerft_CDP2_Export.exe run that includes:
- generation
- sorting
- transform
- compression
- host transfer
- final persistence
- monitor telemetry
- validation against known counts
The historical d9 NB zstd4 export baseline remains:
| Depth | Wall time | Perft throughput | Stored physical rows/sec | Final bytes |
|---|---|---|---|---|
| d9 start position | 242.802 s | 10.047 B/s | 159.904 M/s | 113.397 GB |
The lab result says the next implementation target should not be nvCOMP Cascaded. It should be sorted NB-state rows with ANS and XOR-byteplane, integrated into the actual exporter and measured end to end.
First Real Export Smoke
After the isolated lab, the export executable was updated with:
--terminal-transform byteplane|xor-byteplane
This transform is deliberately final-depth only. Intermediate recurrent scratch files stay in the existing byteplane format so the next-depth reader remains safe. If a transformed file is accidentally used as recurrent input, the reader rejects it instead of silently reading the wrong layout.
The first real-pipeline smoke runs validated start-position export through d6 and d7:
| Run | Depth | Transform | Wall time | Physical rows/sec | Final bytes | Bits/position |
|---|---|---|---|---|---|---|
| 1M parent batch | 6 | byteplane | 0.578 s | 16.292 M | 59.410 MB | 50.470 |
| 1M parent batch | 6 | XOR-byteplane | 0.565 s | 16.664 M | 30.997 MB | 26.330 |
| 131K parent batch | 7 | byteplane | 2.257 s | 96.496 M | 1.488 GB | 54.660 |
| 131K parent batch | 7 | XOR-byteplane | 2.306 s | 94.451 M | 755.045 MB | 27.740 |
Interpretation:
- XOR-byteplane roughly halves bytes written.
- At d6 it slightly improved wall time.
- At d7 it slightly reduced persisted rows/sec because the extra transform work offset the smaller output.
- A 1M parent-batch d7 attempt showed a bad operating shape: VRAM filled, terminal files did not grow, and progress was not observable. Reducing parent batch size to 131K completed and validated. That points to scheduler and chunk-admission work before any larger d9 proof.
The updated conclusion is therefore narrower: ANS plus terminal XOR-byteplane is a useful experimental route and a density win, but it is not yet a proven end-to-end persisted rows/sec win. The next bottleneck is full-pipeline scheduling and write critical-path utilization.
Writer Cadence Follow-Up
The next measurement pass added producer-cadence counters to the direct writer.
This separated active write time from time spent waiting for the next compressed
group to arrive.
The result was clear: the direct writer is mostly idle during the d7 export
critical path. The current bottleneck is not raw disk bandwidth. It is the
cadence at which the exporter produces compressed write-ready groups.
| Variant | Parent chunk | Accum target | Transform | Wall time | Physical rows/sec | Final bytes | Bits/position | Writer idle | Max producer gap |
|---|---|---|---|---|---|---|---|---|---|
| Baseline sync | 131K | 1,048K | byteplane | 2.307 s | 94.395 M | 1.488 GB | 54.657 | 1689.560 ms | 171.686 ms |
| Baseline sync | 131K | 1,048K | XOR-byteplane | 2.371 s | 91.832 M | 755.045 MB | 27.737 | 1758.390 ms | 179.806 ms |
| Sync collapsed | 131K | 1,048K | byteplane | 2.269 s | 95.969 M | 1.488 GB | 54.657 | 1576.560 ms | 170.696 ms |
| Sync collapsed | 262K | 1,048K | byteplane | 2.365 s | 86.670 M | 1.330 GB | 51.904 | 1794.160 ms | 337.551 ms |
| Sync collapsed | 64K | 1,048K | byteplane | 2.524 s | 105.129 M | 1.929 GB | 58.183 | 1797.370 ms | 99.133 ms |
| Sync collapsed | 131K | 524K | byteplane | 2.261 s | 96.313 M | 1.488 GB | 54.657 | 1659.240 ms | 169.865 ms |
| Sync collapsed d8 | 131K | 524K | byteplane | 28.493 s | 85.270 M | 17.869 GB | 58.837 | 26532.990 ms | 132.390 ms |
| Sync collapsed d8 | 262K | 524K | byteplane | 25.860 s | 86.011 M | 15.133 GB | 54.430 | 24401.220 ms | 225.290 ms |
| Default chunk poll | 1,048K | 1,048K | byteplane | killed at 130 s | n/a | n/a | n/a | n/a | no depth-7 progress |
Two useful changes came out of this pass:
- The exporter no longer synchronizes after byteplane transform and again after
nvCOMP compression just to collect timing. Those stages already run in order
on the same CUDA stream, so timing is now collected after the existing metadata
synchronization. This modestly improved the best 131K byteplane d7 run. - The writer report now includes
write_idle_ms,
producer_enqueue_span_ms, andproducer_max_enqueue_gap_ms, making this
bottleneck visible in every future run.
The important negative result is that smaller chunks can increase reported
physical rows/sec by emitting more physical rows, but they also increase total
bytes and wall time. That is diagnostic evidence, not a shipping route.
The d8 confirmation also showed that the d7-safe chunk shape is not acceptable
as a global policy. It landed around 85-86M physical rows/sec, below the
earlier direct ANS d8 neighborhood of about 109M physical rows/sec. The
next policy must be adaptive to current frontier and output size rather than a
fixed chunk setting.
The fixed default chunk shape was retested with polling. It reached the
depth-7 budget line and then produced no new log output or .gpspill growth for
120 seconds, so it was killed at 130 seconds. That confirms the current
wrapper-level “auto” is not really adaptive: it becomes a fixed 1,048K parent
batch before the frontier driver sees the live frontier shape.
The next engineering target is therefore to overlap independent compression
groups or otherwise decouple compressed-group production from the single serial
callback path while preserving exact NB-state aggregation.
The immediate next code target is to move chunk-record auto selection into the
frontier driver and base it on projected child/output width from prior frontier
width and multiplicity growth, bounded by the current GPU/host staging budget.