The Fundamental Law of Edge LLM Serving:
On unified-memory integrated GPUs, compute is effectively free while memory traffic is intensely scarce. Eliminating intermediate DRAM materialization and coalescing memory access matters substantially more than saving integer arithmetic instructions.
80.01 tok/s
AC Power GPU-Only 5-Run Avg (82.77 peak, --vocab 65536 --g128)
95.5%
GEMV DRAM Roofline (15.54 / 16.28 GB/s vector-add)
8.69×
Mean Fusion Speedup (11.34× median reduction)
1.73×
vs. llama.cpp Vulkan (+11.5% full-vocab, +72.5% pruned+g128)

1. The Memory Wall on Unified-Memory iGPUs

Autoregressive LLM token generation (batch size $M = 1$) is fundamentally memory-bandwidth bound. Every single generated token requires streaming the entire model weight matrix through the execution units:

$$\text{Tokens / second} \approx \frac{\text{Memory Bandwidth (GB/s)}}{\text{Model Size (GB)}}$$

1.1 Single-Token Physical Roofline

On integrated GPUs like Intel(R) UHD Graphics (Xe-LP) (Driver 1.4.323), the execution units share system DRAM with the CPU host. To establish unassailable empirical baselines rather than relying on theoretical spec-sheet numbers, we measured the actual hardware limits directly using Vulkan compute timestamps (52.0833 ns tick period) on AC power via bandwidth_benchmark.exe:

  • 18.44 GB/s peak (16.99 GB/s avg) — Pure streaming read roofline (300 MB buffer, 10 runs, 17.056 ms peak / 18.520 ms avg).
  • 13.67 GB/s peak (11.59 GB/s avg) — Device copy bandwidth (128 MB Read+Write).
  • 15.30 GB/s peak (13.94 GB/s avg) — Vector-add bandwidth (192 MB 2R+1W; earlier single-run GEMV baseline measured 16.28 GB/s).

For Qwen2.5-0.5B in Q4_0 INT4 quantization (~278–295 MB per-token weight footprint at full 151,936 vocab), this physical constraint dictates a strict single-token hardware ceiling (~55–62 tok/s). With vocabulary pruning + Group-128 (--vocab 65536 --g128), per-token reads shrink to ~211 MB (4.125 bpw), lifting the AC-power roofline to:

$$\text{Pruned + Group-128 Roofline} \approx \frac{16.99\text{ to }18.44\text{ GB/s}}{0.211\text{ GB}} \approx \mathbf{80.5\text{ to }87.4\text{ tokens/sec}}$$

2. Zero-Framework Architecture (Pure C + GLSL)

Rather than building on high-level runtimes like PyTorch or embedding large frameworks like ggml, we engineered the entire execution system from first principles in C (src/engine/qwen_decode.c, compiled with clang.exe -O3 -nostdlib directly linked against vulkan-1.lib) coupled with 21 custom GLSL compute shaders in shaders/decode/.

2.1 Direct Zero-Copy GGUF Loader

Our lightweight GGUF header parser (src/engine/gguf.h) maps model tensors directly into Vulkan device-local storage buffers with zero host-side staging overhead, ensuring minimal runtime memory footprint.

2.2 24-Layer GPU Pipeline

The autoregressive generation pass executes across 24 Transformer layers entirely on-GPU without CPU synchronization bubbles:

$$\text{Residual} + \text{RMSNorm} \longrightarrow \text{SLM Q4\_0 QKV GEMV} \longrightarrow \text{Dynamic RoPE} \longrightarrow \text{KV Append} \longrightarrow \text{GQA} \longrightarrow \text{Fused SwiGLU + Down}$$

2.3 Zero-Sync 2-Pass GPU Argmax

Standard inference frameworks read the entire logit distribution (151,936 FP16 values = ~304 KB) back to host memory per token. We implemented a 2-pass parallel tree reduction shader on-GPU, transferring exactly 4 bytes (a single uint32_t token ID) back to the CPU per decode step.

src/engine/qwen_decode.c • Vulkan Command Stream Dispatch
// Direct Vulkan compute dispatch loop without CPU-side synchronization
for (uint32_t step = 0; step < target_tokens; ++step) {
    // 1. Dispatch 24 Transformer layers sequentially on the GPU queue
    for (uint32_t layer = 0; layer < NUM_LAYERS; ++layer) {
        vkCmdBindPipeline(cmd, VK_PIPELINE_BIND_POINT_COMPUTE, p_fused_rmsnorm_qkv);
        vkCmdDispatch(cmd, (HIDDEN_DIM + 255) / 256, 1, 1);
        
        vkCmdPipelineBarrier(cmd, VK_PIPELINE_STAGE_COMPUTE_SHADER_BIT,
                             VK_PIPELINE_STAGE_COMPUTE_SHADER_BIT, 0, 0, NULL, 1, &buf_barrier, 0, NULL);
        
        vkCmdBindPipeline(cmd, VK_PIPELINE_BIND_POINT_COMPUTE, p_gqa_decode);
        vkCmdDispatch(cmd, NUM_HEADS, 1, 1);
    }
    
    // 2. 2-Pass On-GPU Argmax over 151,936 vocab tokens (Transfers exactly 1 uint32_t)
    vkCmdBindPipeline(cmd, VK_PIPELINE_BIND_POINT_COMPUTE, p_argmax_pass1);
    vkCmdDispatch(cmd, (VOCAB_SIZE + 1023) / 1024, 1, 1);
    vkCmdBindPipeline(cmd, VK_PIPELINE_BIND_POINT_COMPUTE, p_argmax_pass2);
    vkCmdDispatch(cmd, 1, 1, 1);
}

3. Key Systems Optimizations

3.1 Split uvec4 Group-32 Layout & In-Register Dequantization

GGUF's standard 18-byte block_q4_0 layout interleaves a 2-byte FP16 scale with 16 bytes of packed 4-bit nibbles. This creates unaligned 18-byte stride boundaries that degrade GPU memory bus transactions. We restructured the memory layout into coalesced uvec4 blocks with separate FP16 group-32 scale buffers (gemv_int4_group32.comp). Unpacking 8 nibbles per 32-bit register sustained 15.54 GB/s (95.5% of DRAM ceiling) on $8192 \times 8192$ GEMV—a 3.55× speedup over FP16 (2.431 ms vs. 8.627 ms).

shaders/decode/gemv_int4_group32.comp • In-Register Nibble Dequantization
#version 450
#extension GL_KHR_shader_subgroup_arithmetic : enable

layout(local_size_x = 32, local_size_y = 1) in;

layout(binding = 0) readonly buffer WeightsNibbles { uvec4 w_data[]; };
layout(binding = 1) readonly buffer WeightsScales  { float16_t scales[]; };
layout(binding = 2) readonly buffer InActivation   { vec4 act_data[]; };
layout(binding = 3) writeonly buffer OutVector     { float out_vec[]; };

void main() {
    uint row = gl_GlobalInvocationID.y;
    uint lane = gl_SubgroupInvocationID;
    
    // Unpack 8 INT4 nibbles directly in registers
    uvec4 packed_val = w_data[row * STRIDE + lane];
    vec4 low  = vec4((packed_val.x >> 0) & 0x0Fu, (packed_val.x >> 4) & 0x0Fu,
                     (packed_val.x >> 8) & 0x0Fu, (packed_val.x >> 12) & 0x0Fu) - 8.0;
    vec4 high = vec4((packed_val.x >> 16) & 0x0Fu, (packed_val.x >> 20) & 0x0Fu,
                     (packed_val.x >> 24) & 0x0Fu, (packed_val.x >> 28) & 0x0Fu) - 8.0;

    float scale = float(scales[row * NUM_GROUPS + (lane / 4)]);
    float dot_accum = dot(low * scale, act_data[lane * 2].xyzw) +
                      dot(high * scale, act_data[lane * 2 + 1].xyzw);
                      
    // Coalesced Subgroup Reduction across SIMD32 width
    float row_sum = subgroupAdd(dot_accum);
    if (subgroupElect()) out_vec[row] = row_sum;
}

3.2 SLM Activation Caching & K=896 Specialization

Caching the input activation vector in Shared Local Memory (SLM) ensures that workgroups read the activation from DRAM exactly once, serving subsequent dot products directly from high-speed on-chip SRAM. Specializing gemv_int4_k896_slm.comp for Qwen2.5-0.5B's exact hidden dimension ($K = 896$) unrolls loops completely and removes boundary branching.

3.3 Aggressive Operator Fusion

Fusing residual addition directly with RMSNorm (residual_rmsnorm.comp) and fusing gate_proj + up_proj + SiLU into a single pass (gemv_q4_swiglu_slm.comp) eliminates hundreds of intermediate DRAM roundtrips per token across the entire 24-layer graph.

3.4 N-Gram Self-Speculative Batched Verification

To break past the 1-token-per-pass roofline limit without loading an external draft model into scarce VRAM, our engine uses draft_ngram:

  • Scans recent token history for prefix matches and proposes up to MAX_BATCH = 4 candidate tokens.
  • Batched GEMM and attention shaders verify all candidates in parallel while streaming weights from DRAM only once.
  • Generated 160 tokens in 147 forward passes (1.088 tokens/step, 55 drafted / 12 accepted = 21.8%), raising baseline peak throughput to 59.97 tok/s.

3.5 Bandwidth-Reduction & Drafting Flags (qwen_cfg.txt)

Because decode throughput is strictly proportional to bytes read per token, we added configurable quantization and drafting flags to isolate how far DRAM traffic can be compressed without external draft models:

  • --vocab 65536 --g128 --no-spec (Best Throughput): Prunes the 151,936-token LM head to the top 65,536 tokens (eliminating ~42 MB/token of dead-tail reads) and replaces 32-weight scales with Group-128 scales (4.125 bpw). Per-token reads drop from ~278 MB to ~211 MB, reaching 80.01 tok/s GPU-only (peak 82.77 tok/s, ~16.9 GB/s = 91.6% of peak / 99.5% of avg streaming read) and 77.28 tok/s wall-clock (peak 79.94 tok/s) on AC power.
  • --g128 --no-spec (Full Vocab + Group-128): Keeps all 151,936 vocab logits while shrinking per-token reads to ~255 MB, raising full-vocab speed from 54.20 / 52.18 tok/s to 66.28 tok/s GPU-only / 63.56 tok/s wall-clock.
  • --smart-draft (4..2-gram + Phrase Corpus + 1-gram Fallback): Increases acceptance from 1.08 to 1.17 tok/step (1.42 tok/step with --vocab 65536 --g128), but low acceptance on general chat (12.2%–23.9%) incurs M=4 batched compute overhead on UHD EUs (36.28 / 47.39 tok/s). Thus --no-spec wins on general prose while --code / --json benefit on repetitive syntax.
  • --q3-ffn (3-bit FFN + Group-128, 3.125 bpw): Cuts Layer-0 FFN latency to 0.461 ms (Gate+Up+SwiGLU = 181.51 μs, Down = 103.85 μs, reaching 60.83 wall / 64.21 GPU tok/s full-vocab spec=OFF), though double-quantizing Q4_0 FFN weights to 3-bit without calibration degrades output coherence.

4. Engineering Challenges & Rigorous Methodology

Challenge 1: Thermal & Package-Power Throttling on Thin-and-Light Hardware Back-to-back runs on modern laptop processors share strict CPU/iGPU package power limits (PL1/PL2). In our initial testing, Run 1 delivered 59.97 tok/s GPU-only, while Run 5 dropped to 49.67 tok/s as thermal soaking occurred. Single-run benchmarks are scientifically invalid on such architectures. We instituted a rigorous protocol: 5 consecutive runs reporting mean ± stddev across all evaluations.
Challenge 2: Windows -nostdlib Linkage & High-Resolution Timing Compiling clean C with -nostdlib on Windows required custom CRT stub implementations, explicit _fltused symbols, and hardware QueryPerformanceCounter (QPC) wall-clock clocks synchronized with Vulkan hardware timestamps.
Challenge 3: Distinguishing Physical Roofline from Speculative Acceleration Hitting 59.97 tok/s on a 55.2 tok/s physical memory limit seemed paradoxical until we mathematically isolated the single-token roofline from speculative acceptance rate: $55.2 \times 1.088 = \mathbf{60.1\text{ tokens/sec}}$ effective ceiling.

5. Controlled Microbenchmarks & Ablations (5 Runs, $8192 \times 8192$)

To determine where performance gains originate, we isolated each architectural decision on an $8192 \times 8192$ matrix multiplication (5 warmup + 20 timed iterations per run across 5 consecutive runs):

Ablation Target Variant A (Mean ± Stddev) Variant B (Mean ± Stddev) Empirical Finding
Operator Fusion (INT4) Fused: 3.540 ± 1.635 ms (med 2.713 ms) Unfused 2-Pass: 30.770 ± 1.147 ms 8.69× mean (11.34× median) faster by avoiding 288 MB DRAM roundtrip.
Subgroup Reductions SIMD32 subgroupAdd: 3.540 ± 1.635 ms (min 2.308 ms) SLM Shared Tree: 2.418 ± 0.127 ms (min 2.295 ms) Near parity at minimum latency; reduction time is completely hidden by DRAM latency.
Hardware DP4A (INT8) FP16 Unpack: 4.041 ± 0.074 ms DP4A (dotPacked4x8EXT): 4.085 ± 0.129 ms Identical within 1σ; arithmetic savings cannot overcome the 16.28 GB/s memory bottleneck.

5.4 Precision & Grouped Quantization GEMV Matrix

Measured against the 16.28 GB/s vector-add hardware roofline across $8192 \times 8192$ and $4096 \times 4096$ matrices (all kernels verified PASSED (0.0000) against FP32 reference):

Kernel / Format ($8192 \times 8192$) Footprint Latency Speedup vs FP16 Bandwidth (% Roofline) Rate (inf/s)
FP16 Baseline 128.00 MB 8.627 ms 1.00× 15.56 GB/s (95.6%) 115.9 inf/s
INT8 Fused 64.00 MB 4.522 ms 1.91× 14.85 GB/s (91.2%) 221.1 inf/s
INT4 Per-Row 32.00 MB 2.173 ms 3.97× 15.46 GB/s (95.0%) 460.3 inf/s
Group-32 AWQ (4.5 bpw) 36.00 MB 2.431 ms 3.55× 15.54 GB/s (95.5%) 411.4 inf/s
Native GGUF Q4_0 (4.5 bpw) 36.00 MB 2.484 ms 3.47× 15.21 GB/s (93.4%) 402.5 inf/s

On smaller $4096 \times 4096$ hidden-layer matrices, FP16 (32 MB) runs in 2.745 ms (364.3 inf/s), INT8 Fused (16 MB) drops to 1.296 ms (2.12×, 771.5 inf/s), and INT4 Fused (8 MB) runs in 1.868 ms (535.2 inf/s).

6. End-to-End Results vs. llama.cpp

We benchmarked full autoregressive decoding on Qwen2.5-0.5B-Instruct (Q4_0, generating 160 tokens) across 5 consecutive runs on AC power against llama.cpp (build 2d8a32502 (12360), llama-bench):

End-to-End Generation Rate on AC Power (tg160, tok/s • 5-Run Mean)
Inference Engine & Config Execution Mode 5-Run Mean (AC Power) Peak / Notes vs. llama.cpp Vulkan
vulkan-llm-decode (--vocab 65536 --g128 --no-spec) GPU-Only (Vulkan Compute) 80.01 tok/s (12.50 ms/tok) Peak 82.77 tok/s (70.28 unplugged) +72.5% (1.73×)
vulkan-llm-decode (--vocab 65536 --g128 --no-spec) Wall-Clock End-to-End 77.28 tok/s (12.94 ms/tok) Peak 79.94 tok/s (66.70 unplugged) +66.6% (1.67×)
vulkan-llm-decode (--g128 --no-spec) GPU-Only / Wall (Full Vocab) 66.28 / 63.56 tok/s ~255 MB/token (4.125 bpw) +42.9% / +37.0%
llama.cpp (ngl=0) CPU Host (ggml-cpu-alderlake) 52.26 ± 0.66 tok/s AC Power baseline +12.7%
vulkan-llm-decode (--no-spec, Full Q4_0) GPU-Only / Wall (Apples-to-Apples) 51.71 / 50.47 tok/s Peak 53.89 GPU / 52.67 Wall +11.5% / +8.8%
llama.cpp (ngl=99) Vulkan iGPU Backend 46.38 ± 0.97 tok/s Full 151,936 vocab baseline Baseline

6.1 Quality, Perplexity (--ppl) & Sequence Inflation Evals

To verify whether 65K vocabulary pruning or Group-128 scales degrades generation quality or causes sequence inflation (shorter subwords inflating raw tok/s without increasing real words/s), we evaluated a 149-token mixed tech, biology, and C++ passage (--ppl) plus 160-token generation (--no-spec), including quantizing directly from FP16 (--fp16) instead of double-quantizing Q4_0:

Source → Quant Vocab Pruned OOV PPL (non-OOV) Next-Tok Acc Wall / GPU tok/s Chars/tok Effective Words/s
Q4_0 (full) 151,936 0.0% 160.58 27.5% 51.33 / 52.52 5.58 45.2 w/s
Q4_0 (--vocab 65536) Prefix 65,536 5.4% 154.37 27.5% 52.53 / 53.84 4.95 41.7 w/s
Q4_0 (--vocab 65536 --freq-vocab) Filtered 65,536 2.0% 152.97 27.5% 59.43 / 61.22 4.95 47.2 w/s
Q4_0 → Group-128 (--vocab 65536 --freq-vocab --g128) Filtered 65,536 2.0% 198.15 24.8% 65.85 / 67.74 2.37 30.0 w/s (list drift)
FP16 → Group-32 (--fp16) 151,936 0.0% 158.15 28.9% 61.18 / 64.12 5.28 55.8 w/s
FP16 → Group-128 (--fp16 --g128) 151,936 0.0% 208.68 25.5% 53.83 / 56.35 4.26 — (switched to Chinese)
FP16 → Group-128 (--fp16 --vocab 65536 --freq-vocab --g128) Filtered 65,536 2.0% 193.05 25.5% 68.94 / 71.75 4.89 56.9 w/s (coherent English)
Key Research Takeaways
  • Fair Full-Vocab Q4_0 Comparison: On identical 151,936-vocab Q4_0 weights, qwen_decode.exe reaches 51.71 ± 2.02 tok/s GPU-only (50.47 ± 1.98 wall), outperforming llama.cpp Vulkan (46.38 ± 0.97) by +11.5% while landing right alongside llama.cpp CPU (52.26 ± 0.66) on the shared DRAM bus.
  • Bandwidth Reduction & Sequence Inflation: Filtering to ASCII/English tokens (--freq-vocab) cuts pruned OOVs from 5.4% to 2.0% and delivers a real words/sec gain (47.2 vs 45.2 w/s in Q4_0). Double-quantizing Q4_0 → Group-128 caused list-token drift (30.0 w/s), whereas quantizing directly from FP16 (--fp16 --vocab 65536 --freq-vocab --g128) stayed coherent in English at 68.94 tok/s wall / 71.75 tok/s GPU (56.9 words/s, 337.4 chars/s).
  • 95.5% of Vector-Add GEMV Roofline & 11.34× Fusion Gap: Custom uvec4 group-32 dequantization hits 15.54 / 16.28 GB/s, while avoiding intermediate FP16 DRAM materialization yields an 8.69× mean (11.34× median) speedup.
Link copied to clipboard!