Breaking the Memory Wall: Custom C & Vulkan LLM Inference on Unified-Memory iGPUs
An experimental study of memory-efficient quantized decoding on Intel UHD Graphics (Xe-LP): hitting 95.5% of the GEMV DRAM roofline, reaching 80.01 tok/s GPU-only (82.77 peak) on AC power, and outperforming llama.cpp Vulkan by +11.5% (full vocab) to 1.73× (pruned + Group-128).
1. The Memory Wall on Unified-Memory iGPUs
Autoregressive LLM token generation (batch size $M = 1$) is fundamentally memory-bandwidth bound. Every single generated token requires streaming the entire model weight matrix through the execution units:
$$\text{Tokens / second} \approx \frac{\text{Memory Bandwidth (GB/s)}}{\text{Model Size (GB)}}$$
1.1 Single-Token Physical Roofline
On integrated GPUs like Intel(R) UHD Graphics (Xe-LP) (Driver 1.4.323), the execution units share system DRAM with the CPU host. To establish unassailable empirical baselines rather than relying on theoretical spec-sheet numbers, we measured the actual hardware limits directly using Vulkan compute timestamps (52.0833 ns tick period) on AC power via bandwidth_benchmark.exe:
- 18.44 GB/s peak (16.99 GB/s avg) — Pure streaming read roofline (300 MB buffer, 10 runs,
17.056 mspeak /18.520 msavg). - 13.67 GB/s peak (11.59 GB/s avg) — Device copy bandwidth (128 MB Read+Write).
- 15.30 GB/s peak (13.94 GB/s avg) — Vector-add bandwidth (192 MB 2R+1W; earlier single-run GEMV baseline measured 16.28 GB/s).
For Qwen2.5-0.5B in Q4_0 INT4 quantization (~278–295 MB per-token weight footprint at full 151,936 vocab), this physical constraint dictates a strict single-token hardware ceiling (~55–62 tok/s). With vocabulary pruning + Group-128 (--vocab 65536 --g128), per-token reads shrink to ~211 MB (4.125 bpw), lifting the AC-power roofline to:
$$\text{Pruned + Group-128 Roofline} \approx \frac{16.99\text{ to }18.44\text{ GB/s}}{0.211\text{ GB}} \approx \mathbf{80.5\text{ to }87.4\text{ tokens/sec}}$$
2. Zero-Framework Architecture (Pure C + GLSL)
Rather than building on high-level runtimes like PyTorch or embedding large frameworks like ggml, we engineered the entire execution system from first principles in C (src/engine/qwen_decode.c, compiled with clang.exe -O3 -nostdlib directly linked against vulkan-1.lib) coupled with 21 custom GLSL compute shaders in shaders/decode/.
2.1 Direct Zero-Copy GGUF Loader
Our lightweight GGUF header parser (src/engine/gguf.h) maps model tensors directly into Vulkan device-local storage buffers with zero host-side staging overhead, ensuring minimal runtime memory footprint.
2.2 24-Layer GPU Pipeline
The autoregressive generation pass executes across 24 Transformer layers entirely on-GPU without CPU synchronization bubbles:
$$\text{Residual} + \text{RMSNorm} \longrightarrow \text{SLM Q4\_0 QKV GEMV} \longrightarrow \text{Dynamic RoPE} \longrightarrow \text{KV Append} \longrightarrow \text{GQA} \longrightarrow \text{Fused SwiGLU + Down}$$
2.3 Zero-Sync 2-Pass GPU Argmax
Standard inference frameworks read the entire logit distribution (151,936 FP16 values = ~304 KB) back to host memory per token. We implemented a 2-pass parallel tree reduction shader on-GPU, transferring exactly 4 bytes (a single uint32_t token ID) back to the CPU per decode step.
// Direct Vulkan compute dispatch loop without CPU-side synchronization
for (uint32_t step = 0; step < target_tokens; ++step) {
// 1. Dispatch 24 Transformer layers sequentially on the GPU queue
for (uint32_t layer = 0; layer < NUM_LAYERS; ++layer) {
vkCmdBindPipeline(cmd, VK_PIPELINE_BIND_POINT_COMPUTE, p_fused_rmsnorm_qkv);
vkCmdDispatch(cmd, (HIDDEN_DIM + 255) / 256, 1, 1);
vkCmdPipelineBarrier(cmd, VK_PIPELINE_STAGE_COMPUTE_SHADER_BIT,
VK_PIPELINE_STAGE_COMPUTE_SHADER_BIT, 0, 0, NULL, 1, &buf_barrier, 0, NULL);
vkCmdBindPipeline(cmd, VK_PIPELINE_BIND_POINT_COMPUTE, p_gqa_decode);
vkCmdDispatch(cmd, NUM_HEADS, 1, 1);
}
// 2. 2-Pass On-GPU Argmax over 151,936 vocab tokens (Transfers exactly 1 uint32_t)
vkCmdBindPipeline(cmd, VK_PIPELINE_BIND_POINT_COMPUTE, p_argmax_pass1);
vkCmdDispatch(cmd, (VOCAB_SIZE + 1023) / 1024, 1, 1);
vkCmdBindPipeline(cmd, VK_PIPELINE_BIND_POINT_COMPUTE, p_argmax_pass2);
vkCmdDispatch(cmd, 1, 1, 1);
}
3. Key Systems Optimizations
3.1 Split uvec4 Group-32 Layout & In-Register Dequantization
GGUF's standard 18-byte block_q4_0 layout interleaves a 2-byte FP16 scale with 16 bytes of packed 4-bit nibbles. This creates unaligned 18-byte stride boundaries that degrade GPU memory bus transactions. We restructured the memory layout into coalesced uvec4 blocks with separate FP16 group-32 scale buffers (gemv_int4_group32.comp). Unpacking 8 nibbles per 32-bit register sustained 15.54 GB/s (95.5% of DRAM ceiling) on $8192 \times 8192$ GEMV—a 3.55× speedup over FP16 (2.431 ms vs. 8.627 ms).
#version 450
#extension GL_KHR_shader_subgroup_arithmetic : enable
layout(local_size_x = 32, local_size_y = 1) in;
layout(binding = 0) readonly buffer WeightsNibbles { uvec4 w_data[]; };
layout(binding = 1) readonly buffer WeightsScales { float16_t scales[]; };
layout(binding = 2) readonly buffer InActivation { vec4 act_data[]; };
layout(binding = 3) writeonly buffer OutVector { float out_vec[]; };
void main() {
uint row = gl_GlobalInvocationID.y;
uint lane = gl_SubgroupInvocationID;
// Unpack 8 INT4 nibbles directly in registers
uvec4 packed_val = w_data[row * STRIDE + lane];
vec4 low = vec4((packed_val.x >> 0) & 0x0Fu, (packed_val.x >> 4) & 0x0Fu,
(packed_val.x >> 8) & 0x0Fu, (packed_val.x >> 12) & 0x0Fu) - 8.0;
vec4 high = vec4((packed_val.x >> 16) & 0x0Fu, (packed_val.x >> 20) & 0x0Fu,
(packed_val.x >> 24) & 0x0Fu, (packed_val.x >> 28) & 0x0Fu) - 8.0;
float scale = float(scales[row * NUM_GROUPS + (lane / 4)]);
float dot_accum = dot(low * scale, act_data[lane * 2].xyzw) +
dot(high * scale, act_data[lane * 2 + 1].xyzw);
// Coalesced Subgroup Reduction across SIMD32 width
float row_sum = subgroupAdd(dot_accum);
if (subgroupElect()) out_vec[row] = row_sum;
}
3.2 SLM Activation Caching & K=896 Specialization
Caching the input activation vector in Shared Local Memory (SLM) ensures that workgroups read the activation from DRAM exactly once, serving subsequent dot products directly from high-speed on-chip SRAM. Specializing gemv_int4_k896_slm.comp for Qwen2.5-0.5B's exact hidden dimension ($K = 896$) unrolls loops completely and removes boundary branching.
3.3 Aggressive Operator Fusion
Fusing residual addition directly with RMSNorm (residual_rmsnorm.comp) and fusing gate_proj + up_proj + SiLU into a single pass (gemv_q4_swiglu_slm.comp) eliminates hundreds of intermediate DRAM roundtrips per token across the entire 24-layer graph.
3.4 N-Gram Self-Speculative Batched Verification
To break past the 1-token-per-pass roofline limit without loading an external draft model into scarce VRAM, our engine uses draft_ngram:
- Scans recent token history for prefix matches and proposes up to
MAX_BATCH = 4candidate tokens. - Batched
GEMMand attention shaders verify all candidates in parallel while streaming weights from DRAM only once. - Generated 160 tokens in 147 forward passes (
1.088 tokens/step,55 drafted / 12 accepted = 21.8%), raising baseline peak throughput to 59.97 tok/s.
3.5 Bandwidth-Reduction & Drafting Flags (qwen_cfg.txt)
Because decode throughput is strictly proportional to bytes read per token, we added configurable quantization and drafting flags to isolate how far DRAM traffic can be compressed without external draft models:
--vocab 65536 --g128 --no-spec(Best Throughput): Prunes the 151,936-token LM head to the top 65,536 tokens (eliminating ~42 MB/token of dead-tail reads) and replaces 32-weight scales with Group-128 scales (4.125 bpw). Per-token reads drop from ~278 MB to ~211 MB, reaching 80.01 tok/s GPU-only (peak 82.77 tok/s, ~16.9 GB/s = 91.6% of peak / 99.5% of avg streaming read) and 77.28 tok/s wall-clock (peak 79.94 tok/s) on AC power.--g128 --no-spec(Full Vocab + Group-128): Keeps all 151,936 vocab logits while shrinking per-token reads to ~255 MB, raising full-vocab speed from54.20 / 52.18 tok/sto 66.28 tok/s GPU-only / 63.56 tok/s wall-clock.--smart-draft(4..2-gram + Phrase Corpus + 1-gram Fallback): Increases acceptance from1.08to1.17 tok/step(1.42 tok/stepwith--vocab 65536 --g128), but low acceptance on general chat (12.2%–23.9%) incursM=4batched compute overhead on UHD EUs (36.28 / 47.39 tok/s). Thus--no-specwins on general prose while--code/--jsonbenefit on repetitive syntax.--q3-ffn(3-bit FFN + Group-128,3.125 bpw): Cuts Layer-0 FFN latency to0.461 ms(Gate+Up+SwiGLU = 181.51 μs,Down = 103.85 μs, reaching 60.83 wall / 64.21 GPU tok/s full-vocabspec=OFF), though double-quantizingQ4_0FFN weights to 3-bit without calibration degrades output coherence.
4. Engineering Challenges & Rigorous Methodology
59.97 tok/s GPU-only, while Run 5 dropped to 49.67 tok/s as thermal soaking occurred. Single-run benchmarks are scientifically invalid on such architectures. We instituted a rigorous protocol: 5 consecutive runs reporting mean ± stddev across all evaluations.
-nostdlib Linkage & High-Resolution Timing
Compiling clean C with -nostdlib on Windows required custom CRT stub implementations, explicit _fltused symbols, and hardware QueryPerformanceCounter (QPC) wall-clock clocks synchronized with Vulkan hardware timestamps.
59.97 tok/s on a 55.2 tok/s physical memory limit seemed paradoxical until we mathematically isolated the single-token roofline from speculative acceptance rate: $55.2 \times 1.088 = \mathbf{60.1\text{ tokens/sec}}$ effective ceiling.
5. Controlled Microbenchmarks & Ablations (5 Runs, $8192 \times 8192$)
To determine where performance gains originate, we isolated each architectural decision on an $8192 \times 8192$ matrix multiplication (5 warmup + 20 timed iterations per run across 5 consecutive runs):
| Ablation Target | Variant A (Mean ± Stddev) | Variant B (Mean ± Stddev) | Empirical Finding |
|---|---|---|---|
| Operator Fusion (INT4) | Fused: 3.540 ± 1.635 ms (med 2.713 ms) |
Unfused 2-Pass: 30.770 ± 1.147 ms |
8.69× mean (11.34× median) faster by avoiding 288 MB DRAM roundtrip. |
| Subgroup Reductions | SIMD32 subgroupAdd: 3.540 ± 1.635 ms (min 2.308 ms) |
SLM Shared Tree: 2.418 ± 0.127 ms (min 2.295 ms) |
Near parity at minimum latency; reduction time is completely hidden by DRAM latency. |
| Hardware DP4A (INT8) | FP16 Unpack: 4.041 ± 0.074 ms |
DP4A (dotPacked4x8EXT): 4.085 ± 0.129 ms |
Identical within 1σ; arithmetic savings cannot overcome the 16.28 GB/s memory bottleneck. |
5.4 Precision & Grouped Quantization GEMV Matrix
Measured against the 16.28 GB/s vector-add hardware roofline across $8192 \times 8192$ and $4096 \times 4096$ matrices (all kernels verified PASSED (0.0000) against FP32 reference):
| Kernel / Format ($8192 \times 8192$) | Footprint | Latency | Speedup vs FP16 | Bandwidth (% Roofline) | Rate (inf/s) |
|---|---|---|---|---|---|
| FP16 Baseline | 128.00 MB | 8.627 ms | 1.00× | 15.56 GB/s (95.6%) | 115.9 inf/s |
| INT8 Fused | 64.00 MB | 4.522 ms | 1.91× | 14.85 GB/s (91.2%) | 221.1 inf/s |
| INT4 Per-Row | 32.00 MB | 2.173 ms | 3.97× | 15.46 GB/s (95.0%) | 460.3 inf/s |
Group-32 AWQ (4.5 bpw) |
36.00 MB | 2.431 ms | 3.55× | 15.54 GB/s (95.5%) | 411.4 inf/s |
Native GGUF Q4_0 (4.5 bpw) |
36.00 MB | 2.484 ms | 3.47× | 15.21 GB/s (93.4%) | 402.5 inf/s |
On smaller $4096 \times 4096$ hidden-layer matrices, FP16 (32 MB) runs in 2.745 ms (364.3 inf/s), INT8 Fused (16 MB) drops to 1.296 ms (2.12×, 771.5 inf/s), and INT4 Fused (8 MB) runs in 1.868 ms (535.2 inf/s).
6. End-to-End Results vs. llama.cpp
We benchmarked full autoregressive decoding on Qwen2.5-0.5B-Instruct (Q4_0, generating 160 tokens) across 5 consecutive runs on AC power against llama.cpp (build 2d8a32502 (12360), llama-bench):
| Inference Engine & Config | Execution Mode | 5-Run Mean (AC Power) | Peak / Notes | vs. llama.cpp Vulkan |
|---|---|---|---|---|
vulkan-llm-decode (--vocab 65536 --g128 --no-spec) |
GPU-Only (Vulkan Compute) | 80.01 tok/s (12.50 ms/tok) | Peak 82.77 tok/s (70.28 unplugged) | +72.5% (1.73×) |
vulkan-llm-decode (--vocab 65536 --g128 --no-spec) |
Wall-Clock End-to-End | 77.28 tok/s (12.94 ms/tok) | Peak 79.94 tok/s (66.70 unplugged) | +66.6% (1.67×) |
vulkan-llm-decode (--g128 --no-spec) |
GPU-Only / Wall (Full Vocab) | 66.28 / 63.56 tok/s | ~255 MB/token (4.125 bpw) | +42.9% / +37.0% |
llama.cpp (ngl=0) |
CPU Host (ggml-cpu-alderlake) |
52.26 ± 0.66 tok/s | AC Power baseline | +12.7% |
vulkan-llm-decode (--no-spec, Full Q4_0) |
GPU-Only / Wall (Apples-to-Apples) | 51.71 / 50.47 tok/s | Peak 53.89 GPU / 52.67 Wall | +11.5% / +8.8% |
llama.cpp (ngl=99) |
Vulkan iGPU Backend | 46.38 ± 0.97 tok/s | Full 151,936 vocab baseline | Baseline |
6.1 Quality, Perplexity (--ppl) & Sequence Inflation Evals
To verify whether 65K vocabulary pruning or Group-128 scales degrades generation quality or causes sequence inflation (shorter subwords inflating raw tok/s without increasing real words/s), we evaluated a 149-token mixed tech, biology, and C++ passage (--ppl) plus 160-token generation (--no-spec), including quantizing directly from FP16 (--fp16) instead of double-quantizing Q4_0:
| Source → Quant | Vocab | Pruned OOV | PPL (non-OOV) | Next-Tok Acc | Wall / GPU tok/s | Chars/tok | Effective Words/s |
|---|---|---|---|---|---|---|---|
Q4_0 (full) |
151,936 | 0.0% | 160.58 | 27.5% | 51.33 / 52.52 | 5.58 | 45.2 w/s |
Q4_0 (--vocab 65536) |
Prefix 65,536 | 5.4% | 154.37 | 27.5% | 52.53 / 53.84 | 4.95 | 41.7 w/s |
Q4_0 (--vocab 65536 --freq-vocab) |
Filtered 65,536 | 2.0% | 152.97 | 27.5% | 59.43 / 61.22 | 4.95 | 47.2 w/s |
Q4_0 → Group-128 (--vocab 65536 --freq-vocab --g128) |
Filtered 65,536 | 2.0% | 198.15 | 24.8% | 65.85 / 67.74 | 2.37 | 30.0 w/s (list drift) |
FP16 → Group-32 (--fp16) |
151,936 | 0.0% | 158.15 | 28.9% | 61.18 / 64.12 | 5.28 | 55.8 w/s |
FP16 → Group-128 (--fp16 --g128) |
151,936 | 0.0% | 208.68 | 25.5% | 53.83 / 56.35 | 4.26 | — (switched to Chinese) |
FP16 → Group-128 (--fp16 --vocab 65536 --freq-vocab --g128) |
Filtered 65,536 | 2.0% | 193.05 | 25.5% | 68.94 / 71.75 | 4.89 | 56.9 w/s (coherent English) |
- Fair Full-Vocab
Q4_0Comparison: On identical 151,936-vocabQ4_0weights,qwen_decode.exereaches 51.71 ± 2.02 tok/s GPU-only (50.47 ± 1.98wall), outperformingllama.cppVulkan (46.38 ± 0.97) by +11.5% while landing right alongsidellama.cppCPU (52.26 ± 0.66) on the shared DRAM bus. - Bandwidth Reduction & Sequence Inflation: Filtering to ASCII/English tokens (
--freq-vocab) cuts pruned OOVs from5.4%to2.0%and delivers a real words/sec gain (47.2 vs 45.2 w/sinQ4_0). Double-quantizingQ4_0 → Group-128caused list-token drift (30.0 w/s), whereas quantizing directly from FP16 (--fp16 --vocab 65536 --freq-vocab --g128) stayed coherent in English at 68.94 tok/s wall / 71.75 tok/s GPU (56.9 words/s, 337.4 chars/s). - 95.5% of Vector-Add GEMV Roofline & 11.34× Fusion Gap: Custom
uvec4group-32 dequantization hits15.54 / 16.28 GB/s, while avoiding intermediate FP16 DRAM materialization yields an8.69×mean (11.34×median) speedup.