Parallel is not concurrency
Contents
TL;DR: --parallel sets a slot count. It does not make the concurrency. A named count cuts the KV cache into fixed parts. A llama.cpp server with no flag opens four slots at the full context. The flag costs 662 MiB. If you add --kv-unified, you keep the count and the shared pool.
✅ The server is concurrent before you set a flag. The boot line with no --parallel shows n_slots = 4, n_ctx_slot = 32768, kv_unified = 'true'. There are four slots and one shared pool. Each slot can use all 32,768 tokens.
✅ --parallel 2 gives fewer slots. The same model at the same context shows n_slots = 2, n_ctx_slot = 32768, kv_unified = 'false'. There are two slots, and the slots do not share the pool.
✅ The flag changes the cache layout. The KV cache holds the memory of one conversation. It holds one key and one value for each token in each layer. The server takes one block of memory at start. --ctx-size sets the size of the block. --parallel sets the slot count. A named count cuts the block into fixed parts.
🔴 Caution: --ctx-size sets the full pool, not one conversation. --ctx-size 32768 --parallel 2 gives n_ctx_slot = 16384. The server prints no warning. If you want 32k for each of two slots, ask for 65,536.
🔴 Caution: --parallel 1 is different from no flag. You get one slot, and all other requests wait. This setting looks safe, but it is the most strict setting.
✅ You can have a count and a shared pool. The count and the cut are two switches. --parallel 8 --kv-unified gives eight slots at 32,768 tokens from one pool. It costs 25,495 MiB. The four-slot default costs 25,492 MiB. The slot count is free. The pool size is the cost.
I set --parallel 2 to get a server for two requests at the same time. The server already did this. The flag cut the conversation memory into two fixed halves. It also took 662 MiB for the half that I do not use. The word parallel does not tell you how many people can speak to the model. It tells you how the server cuts one block of memory.
What the flag looks like it does
Each inference server has a slot count. Most people read a slot as a worker. One slot takes one request. Two slots take two requests. With that model in mind, you increase the number when more users arrive. You pay a small quantity of memory for it.
That model is wrong, and the error is expensive. The server shows this on the first line at boot.
The line that shows the truth
The llama.cpp server prints this line when it loads a model:
load_model: initializing, n_slots = 4, n_ctx_slot = 32768, kv_unified = 'true'
This is the boot line with no --parallel flag. There are four slots. Each slot holds a conversation of 32,768 tokens. One shared pool supplies all four slots. kv_unified = true shows this condition.
The same model at the same context with --parallel 2 prints this line:
load_model: initializing, n_slots = 2, n_ctx_slot = 32768, kv_unified = 'false'
There are fewer slots. The slots no longer share the pool.
The KV carving rule
The rule is short:
--parallelcuts the cache. It does not make the concurrency.
The memory in this rule is the KV cache. The model reads your conversation. For each word in each layer it calculates two numbers. The names of the two numbers are key and value. The model keeps the numbers and does not repeat the calculation for each later word. This is the cache. The cache is not the weights of the model. The cache is the memory of one conversation, and it becomes larger with each token.
The server takes one block of memory at start. --ctx-size sets the size of the block. Two switches then control the block. --parallel sets the slot count. --kv-unified sets the layout: one shared pool, or one fixed part for each slot. The defaults connect the two switches. The shared pool is the default while the slot count stays at auto. If you name a count, you lose the shared pool.
One pool and four straws. With no flag, four slots drink from one pool. Each slot can drink to the bottom. Thus one long conversation can use all of the context. Four short conversations share the pool.
Four cups, one for each slot. With --parallel 4, the server puts the pool into four cups. Each slot has one cup. A slot cannot drink from the cup of a different slot. This rule stays true when the other cup is full and your cup is empty.
The two layouts are equally correct. They answer different questions. The pool answers this question: can all users usually get sufficient context? The cups answer this question: does each user keep a guaranteed share?
The measured cost
These are the properties of the model in the test. It has 24 billion parameters. It uses a mixture of experts. The quantization is 8-bit. It has 40 layers, and 10 layers use attention. Each attention layer has 8 key/value heads. Each head has 64 numbers. Each entry uses 2 bytes.
Calculate the memory for one token:
2 (key and value)
x 10 attention layers
x 8 heads
x 64 numbers per head
x 2 bytes
= 20,480 bytes, or 20 KB per token
One token costs 20 KB. A context of 32,768 tokens costs approximately 640 MiB. This quantity applies to the pool, not to each slot.
The two configurations below give 32,768 tokens to each request:
| Setup | Slots | Pool asked for | Card used |
|---|---|---|---|
| No flag, shared pool | 4 | 32,768 tokens | 25,548 MiB |
--parallel 2, partitioned | 2 | 65,536 tokens | 26,210 MiB |
The measured difference is 662 MiB. The calculation predicts 640 MiB. The difference is a second pool. It buys a reservation for a second slot. On this card it also gives fewer usable slots than the default.
The trap that removes your context
Caution: read this before you set a slot count. --ctx-size sets the full pool. It does not set the size of one conversation. The server divides the pool by the slot count.
If you set --ctx-size 32768 --parallel 2, you get this result:
n_slots = 2, n_ctx_slot = 16384
Each slot gets 16,384 tokens. This is one half of the expected value, and the server gives no warning. If you want 32,768 tokens for each of two slots, set --ctx-size 65536.
Trap: a dashboard can show the context as a value for each request. If it does this, it must multiply the value by the slot count before it calls the server. If it does not multiply, each user gets less context than the interface shows.
--parallel 1 is not the same as no flag
This setting catches careful users. If you set the flag to 1, you get one slot. All other requests wait in a queue. If you do not set the flag, you get four slots and one shared pool. The setting that looks safe is the most strict setting.
How to set the slot count and keep the shared pool
Most users want this configuration. It needs two flags.
--parallel Nsets the slot count. The default is-1, which means auto.--kv-unified(-kvu) sets the shared pool. The default is on while the slot count stays at auto.
Set both flags to get both results:
--ctx-size 32768 --parallel 8 --kv-unified
load_model: initializing, n_slots = 8, n_ctx_slot = 32768, kv_unified = 'true'
There are eight slots. Each slot can use all 32,768 tokens. One pool supplies all eight. The measured cost is 25,495 MiB. The four-slot default costs 25,492 MiB. The two values are the same within the measurement noise of this card, which is approximately 60 MiB.
This is the rule that the numbers give you: the slot count is free, and the pool size is the cost. Eight slots cost the same as four. Two fixed parts cost 718 MiB more, because the parts need a pool of 65,536 tokens.
All figures in this article are a card read minus the idle baseline of the same card. A raw card read also holds the other processes on the card, and a comparison of raw reads makes a difference appear where there is none.
The four conditions are these:
| You pass | Slots | Pool | Behaviour |
|---|---|---|---|
| nothing | auto, 4 here | shared | flexible, cheapest, no guarantees |
--parallel 1 | 1 | one slot | strictly serial, requests queue |
--parallel N | N | split N ways | reserved share each, costs N pools |
--parallel N --kv-unified | N | shared | N slots at full context, one pool’s memory |
Procedure: set the slots
- Read the boot line.
n_slots,n_ctx_slotandkv_unifiedshow the result. Do not trust the flag that you sent. - Calculate the cost of one token with this formula:
2 x attention layers x kv heads x head dimension x bytes per element. Multiply the result by your context to get the pool size. - If you use the server alone, or the traffic comes in bursts, do not set
--parallel. You get four slots, one pool and the lowest memory. - If you want a specified slot count and no fixed parts, set
--parallel Nand--kv-unified. - If each user must get a guaranteed window, set
--parallel Nand do not set--kv-unified. Set the size tocontext x N, and make sure that the memory for N pools is available. - Compare
n_ctx_slotwith the window that you show to your users. The silent division is visible here.
The one line to remember
The flag does not add a lane to the road. It paints lines on the lane that you have. --kv-unified removes the lines again.
Sources
- llama.cpp server documentation - flag reference for
--parallel,--ctx-sizeand--cont-batching - llama.cpp repository - slot handling and unified KV cache implementation