Qwen3.6-35B-A3B at full 256K context on a single 24GB GPU

The new Qwen3.6-35B-A3B fits its full 256K context on a single 24GB card with llama.cpp — and the surprise is that the KV cache stays at plain f16. No cache quantization, no CPU expert-offload. Two things carry it: a 4-bit weight quant, and Qwen3.6’s hybrid linear attention.

The stack: Qwen3.6-35B-A3B-UD-IQ4_XS (~4.25 bpw, ~19 GB weights) · all layers on GPU · flash-attn on · f16 KV cache · 262,144-token context.

Why it fits

IQ4_XS shrinks the weights. 35B at ~4.25 bits/weight lands near 19 GB — that alone is most of a 24 GB budget spent, but it fits.
Hybrid attention keeps the KV cache tiny. On a normal dense 35B, an unquantized KV cache at 256K would be tens of gigabytes and OOM the card instantly. Qwen3.6 runs mostly gated-linear (Mamba-style) attention layers that carry a constant-size state, with only a handful of full-attention layers. The proof is right in the config: they run KV at raw f16 and it still clears 24 GB.
3B active of 35B. Only ~3B parameters fire per token, so throughput stays usable even with the whole model resident.

Counter-intuitive: you do not need KV-cache quantization (turbo / q8_0 / q4_0) or --n-cpu-moe expert-offload here. The architecture already flattens the long-context cost — those levers are only for going bigger or onto a smaller card.

The config (llama.cpp, 24GB GPU)

services:
  llamacpp:
    container_name: llamacpp-qwen3-6-35b-a3b-iq4xs
    image: ghcr.io/ggml-org/llama.cpp:server-cuda
    restart: unless-stopped
    gpus: all
    shm_size: "8gb"
    ipc: host
    environment:
      - NVIDIA_VISIBLE_DEVICES=all
      - NVIDIA_DRIVER_CAPABILITIES=compute,utility
    command:
      - -m
      - /models/Qwen3.6-35B-A3B-UD-IQ4_XS.gguf
      - --host
      - 0.0.0.0
      - --port
      - "8000"
      - --alias
      - qwen3.6-35b-a3b-iq4xs
      - --ctx-size
      - "262144"          # full 256K context
      - --n-gpu-layers
      - "999"             # everything on the GPU, no CPU offload
      - --parallel
      - "1"
      - --threads
      - "8"
      - --flash-attn
      - "on"
      - --batch-size
      - "256"
      - --ubatch-size
      - "256"
      - --cache-type-k
      - f16               # KV stays f16 — hybrid attention keeps it small
      - --cache-type-v
      - f16
      - --temp
      - "0.6"             # Qwen3.6 recommended sampling
      - --top-p
      - "0.95"
      - --top-k
      - "20"
      - --min-p
      - "0.0"
      - --presence-penalty
      - "0.0"
      - --repeat-penalty
      - "1.0"
    volumes:
      - ./models:/models:ro
    ports:
      - "9998:8000"

Grab the weights with huggingface-cli download unsloth/Qwen3.6-35B-A3B-GGUF UD-IQ4_XS/* --local-dir ./models, point -m at the .gguf, then docker compose up -d. OpenAI-compatible endpoint on :9998.

Notes

Headroom scales with VRAM. On a 32 GB card the same recipe runs with room to raise --parallel or the batch size; drop to a smaller card and you’d reach for KV-cache quant or expert-offload to claw it back.
Prefer FP8 or NVFP4 on bigger boxes. On a 128 GB unified-memory machine (e.g. a GB10 Spark), skip the GGUF and serve Qwen3.6-35B-A3B-FP8 under vLLM — the memory is there and you keep full precision. On Blackwell, NVFP4 beats FP8.
Exact VRAM and tokens/s aren’t measured here — the recipe is confirmed working by the community; the numbers are quant math, not a benchmark on this exact card.

Source: r/LocalLLaMA — running Qwen3.6-35B-A3B at full context on a 4090 and a GB10 Spark.