Jordan Anderson

OpenVINO Intel GPU plugin—__local-pointer kernel compilation fixes + shared-header API extension

openvinotoolkit/openvino · 2026

• Worked through a systemic OpenCL compile failure across the GPU plugin’s LoRA, MoE, and fully-connected kernels on Intel Compute Runtime 23.x+—the __local-pointer overloads of intel_sub_group_block_read* had been dropped from the base cl_intel_subgroups extension and now require cl_intel_subgroup_local_block_io, which most current Intel discrete GPUs (Arc, Battlemage, Lunar Lake) don’t advertise. Anyone running LoRA or MoE inference on that hardware was hitting compile errors.
• Verified the substitution against the kernels’ sub-group attributes, SLM barrier ordering, and per-lane distribution semantics; tested end-to-end via a binary-patch of the shipped plugin (same-length replacement strings, bit-equivalent to a source rebuild) before any source PR.
• Designed the API extension to the shared sub_group_block_read.cl header—new __local-pointer overload family + companion macro + two-tier dispatch—migrated four kernels onto it across two PRs, surveyed the plugin for remaining call sites, and retrofitted the original PR’s kernel onto the shared API on reviewer request. Design approved by the subsystem architect and merge gatekeeper; validated on real Arc-class hardware via Intel’s DG2 internal CI.
• Followed up with a regression test that catches the bug class on any driver—the kernel calls the new _sub_group_block_read_slm* family directly, with the standard batch headers wired through the kernels_cache constructor (architect’s preferred pattern)—so pre-fix the symbols don’t exist in either dispatch branch and the build fails everywhere, while post-fix it compiles cleanly. The bug class can’t silently recur in CI.

Impact

Measured on one affected part—a Gen 9.5 UHD P630 on Compute Runtime 23.43, TinyLlama-1.1B int4, 2026-05-21: on the release wheel the GPU path dies in clBuildProgram (the bug itself, caught three times in the log); on the fixed build the GPU decodes at 18.4 tok/s against 13.9 on the CPU fallback (1.33×), with time-to-first-token 133 ms against 355 (2.7×). Energy is unmeasured on that box. The ranges below remain projections for Arc-class and Lunar Lake parts, vs CPU fallback under AUTO/HETERO.

• Latency ~2–5× faster decode, up to ~10× faster TTFT on long prompts, by keeping inference on the GPU instead of falling back to CPU.
• Energy ~2–10× better energy-per-token (largest on low-power iGPUs like Lunar Lake).
• I/O in mixed-execution (HETERO) configs, removes the GPU↔CPU boundary tax (per-crossing sync + kernel-launch + layout-repack) that does no useful arithmetic, by keeping the LoRA/MoE/GEMV kernels resident on the GPU.

• PR #35661 openvinotoolkit/openvino #35661 —merged
• PR #35712 openvinotoolkit/openvino #35712 —merged
• PR #36017 openvinotoolkit/openvino #36017 —merged
• PR #36543 openvinotoolkit/openvino #36543 —core input-validation sweep; approved, unmerged
• Profile github.com/pjordanandrsn

grouped-nf4-gemm & experts-4bit—a fused mixture-of-experts inference kernel, and a preregistration discipline for the claims made about it

pjordanandrsn/grouped-nf4-gemm · experts4bit-qlora · 2026

• A Triton kernel that runs mixture-of-experts matmuls directly on the 4-bit-packed weights—one launch for all active experts, the codebook decoded in registers inside the GEMM, no dequantize-to-bf16 round trip. It consumes the bitsandbytes packed layout unchanged, so it drops into the stack the ecosystem already runs rather than adding a parallel format. The companion library, experts-4bit, is the upstream side of the same idea—4-bit quantization of the fused expert weights that stock bitsandbytes skips—submitted to bitsandbytes as a deliberately clean, reviewable diff.
• The demonstration is a 235-billion-parameter model streamed from system RAM over PCIe with the fused kernel as the only expert compute. The registered run (synthetic expert weights, real attention and router) streams the full 4-bit expert set from about 128 GB of pinned host memory at 5.57 tokens per second, 102–103 percent of the measured link’s waterfall ceiling on a 13.6 GB working set, where the dequantize-then-matmul path on the identical pipeline reaches 1.81. The real 438 GB checkpoint, quantized as it streams, decodes coherent text at 4.3–4.4 tokens per second on 15.2 GB of video memory across five machines. What fraction of the bus a given box reaches turned out not to be a transferable constant: a later run on faster hardware reached a smaller fraction than slower hosts did, which falsified the pure-bandwidth model and forced a published correction adding the per-host serialization term it had omitted—per-token time is bytes over link speed plus a fixed per-host cost, measured between 54 and 114 ms across seven hosts.
• Since July the kernel serves both sixteen-entry codebooks: NF4, and MXFP4—the format frontier labs now ship weights in—computing on a checkpoint’s exact released bytes. That deletes the requantization tax (a measured +9.4 percent perplexity on the 120-billion-parameter case) rather than tolerating it, and it makes the served weights verifiable: an executable receipt re-hashes the model’s byte ranges against the running arena, and a 120-billion-parameter fine-tune closes with every hash identical before and after training, at 9.82 GB of peak video memory. The companion library grew the serving side to match—hot experts resident, the cold tail streamed or computed on the host, hot sets chosen from measured routing rather than by index. Its serving stack now decodes a 30-billion-parameter mixture-of-experts model single-stream at 238 tokens per second on one RTX 5090, with GPTQ-calibrated int4 experts, 2.07× its own NF4 arm on the same card.
• The kernel has been differentiable since the end of July, which is the part with the most practical consequence: fine-tuning computes on the packed 4-bit weights instead of unpacking them in the backward pass, and the backward is one launch, on by default. Across two thirty-billion-parameter-class mixture-of-experts models and five datasets, 200 steps per cell, the fused training path runs 1.5–1.8× faster per step at three-quarters to four-fifths of the peak video memory, inside the registered loss-parity band on every cell, with the frozen 4-bit stack verified byte-identical over 16.31 GB hashed—under a control that flips a byte on purpose to prove the check can fail. Inside a real fine-tune on prose it runs 4.5× the per-expert loop on a 4090 and 4.75× on an H100. The instructive part was not the calculus. Three separate defects hid behind gates that could not fail: a bit-exactness check comparing empty tensors, two consecutive fixes that were never actually in the code under test, and a missing scaling factor that halved every update while remaining invisible to memory, hashes, and single-step comparison alike.
• The part that transfers to a client engagement is the adjudication. The claims distinguish a public measurement from independent confirmation. For the preregistered kernel benchmarks, the protocols and Bitcoin timestamp proofs let a reader check when the pass/fail bars were fixed. Other results retain their measured or measured-private tier; a measurement is not automatically a confirmed result. The failures are published at the same volume as the wins: a split-K variant of the decode kernel refuted and shipped dormant as the evidence, an expert-prefetch program closed negative over four registered arcs, and the finding that against a CUDA-graphed baseline the fused path loses at decode on both a 4090 and an H100—what survives graphing is the memory-traffic win at training shape on bandwidth-limited cards. It is the performance-claim equivalent of a signed advisory: check the work, don’t trust the claimant.

• Kernel github.com/pjordanandrsn/grouped-nf4-gemm
• Library github.com/pjordanandrsn/experts4bit-qlora
• Writeup the method, at length — cerinamroth.com —my research practice
• Upstream bitsandbytes #1965 —in review
• Upstream bitsandbytes #1999 —merged; the parametrize cache leaked under activation checkpointing
• Upstream transformers #47931 —merged; assisted decoding ran past an EOS accepted mid-block
• Upstream unsloth-zoo #1082 —merged; the MoE weight-preprocessor registry resolved through the package

Friction by Other Means—how regulatory infrastructure preserved tariff effects after IEEPA fell

Essay · 2026 · ~7,600 words

The Trump II IEEPA tariff regime collapsed at the Supreme Court on February 20, 2026, but its practical effects largely did not. Tariff revenue, by Treasury’s own projection, will be “virtually unchanged” in 2026. Import friction—CBP audit pace, CF-29 issuance, first-sale challenges, transshipment enforcement, FDA entry-review centralization, de minimis closure, TTB queue lengthening—is at or above the IEEPA-era peak. Refunds are slow-walked through technical capacity constraints layered onto a narrow legal interpretation.

The strongest reading of the evidence is structural convergence with intentional ratification. Most of the infrastructure (CEEs, ACE, PREDICT, FSMA, FAA Act permitting, CBP audit authority) long predates the administration and has its own bureaucratic logic, but the timing, sequencing, public framing, and refund-mechanism design in 2025–2026 indicate the administration recognized the alignment and chose to operationalize it. The durable variable in modern American tariff policy is not the headline rate but the regulatory infrastructure that surrounds it.

• Essay writing/friction-by-other-means

The Structural Failure of America’s Defense Industrial Base — property, profit, and the production crisis

Essay · 2026 · ~4,200 words

The United States placed its war-fighting capacity inside publicly traded corporations whose legal obligations run to shareholders, not to readiness. In 2024, seven major defense primes returned 98 percent of free cash flow to investors. Surge capacity—idle factories, warm production lines, redundant suppliers—generates no shareholder return and gets cut. Operation Epic Fury exposed the result: 800 Patriot interceptors fired in five days against an annual production of 600; 168 Tomahawks fired in the first 100 hours against five-year purchases of 322; one carrier covering the Pacific while the fleet rebalances to the Persian Gulf.

The defense industrial base does not atrophy despite the market working correctly. It atrophies because the market is working correctly, toward an objective that has nothing to do with national security. The structural alternative is the Government-Owned, Contractor-Operated model—proven across 73 facilities in World War II and still operational for nuclear weapons production — which retains public ownership of productive capacity while leveraging private operational expertise.

• Essay writing/the-structural-failure-of-the-defense-industrial-base

The Great Recentralization—civilizational risk in the cloud-AI transition, 2026–2046

Essay · 2026 · ~9,000 words

The locus of meaningful computation has migrated from devices owned and controlled by users to a small set of hyperscale data centers operated by five American firms and one Taiwanese fabricator. Five threat vectors compound: economic concentration exceeding the railroad and oil trusts; surveillance infrastructure beyond what the Stasi could operate; supply-chain fragility centered on a single fab on a contested island; documented atrophy of cognitive capacity in populations that offload reasoning; and the disappearance of the off-grid-capable personal computer.

The cloud-AI transition is the enclosure of compute, of model weights, of the digital commons of personal data, and ultimately of cognition itself. The historical pattern is that enclosure, once accomplished, is not reversed by markets, only by politics. Existing antitrust and right-to-repair instruments regulate the symptoms of platform power while the underlying enclosure of compute proceeds unimpeded.

• Essay writing/the-great-recentralization