SettledOpen source
vLLM adds push-based KV transfer, cutting time to first token up to 3x
AWS engineers contributed a push-based NIXL KV connector to vLLM for disaggregated inference. In AWS's test on Qwen3-32B with 16k inputs at 8 QPS, time to first token fell from about 1,744 ms to 540 ms, and per-token latency fell up to about 30%. One model and one topology were tested, with prefix caching off.
- HEAT
- OUTLETS
- 0
- FIRST SEEN
- LAST SIGNAL
Primary source
AWSaws.amazon.com/blogs/machine-learning/push-based-kv-transfer-to-improve-ttft-and-tpot-for-disaggregated-inference-in-vllm/Coverage · 1 article, oldest first
Monday, October 19, 2026