Asynchronicity in Continuous Batching
Synchronous batching in LLM inference causes inefficiencies by making CPU and GPU work alternately, leading to significant idle time. Asynchronous batching allows CPU and GPU to operate in parallel, improving GPU utilization and reducing overall inference time. This approach requires careful coordination of hardware and data flow but can yield substantial performance gains without modifying models or kernels.
- ▪Synchronous batching results in the CPU and GPU taking turns, leaving the GPU idle for nearly 24% of total runtime.
- ▪Asynchronous batching enables parallel CPU batch preparation and GPU computation, reducing wasted compute time.
- ▪The implementation relies on CUDA streams and events to manage concurrency and ensure data readiness between CPU and GPU tasks.
- ▪Eliminating CPU-induced delays could reduce generation time by 24%, from 300.6 seconds to approximately 228 seconds for 8K token generation.
- ▪No changes to model architecture or kernels are needed to implement asynchronous batching, only proper hardware coordination.
Hacker News (Newest) files mainly under programming. We currently carry 5,306 of its stories.
Story provenance
Source · retrieval · rights · ranking — open for full record
inspect →
Story provenance
Attribution is not the same as permission. This drawer separates discovery metadata, excerpts, WeSearch-generated summaries, reuse status, and whether the publisher receives the visit. Nothing here claims a legal grant the publisher has not made.
Record
| Original publisher | Huggingface |
| Canonical URL | https://huggingface.co/blog/continuous_async |
| Publication time | Sat, 16 May 2026 06:38:38 +0000 |
| Retrieval time | 2026-05-16T07:10:17.660Z |
| Last seen | 2026-05-16T07:10:17.660Z |
| Headline source | Publisher (no WeSearch rewrite) |
| Excerpt source | publisher body |
| Excerpt method | First ~120 words (~800 chars) of extracted publisher body, fair-use limited. |
| Summary | WeSearch · cerebras-chat (WeSearch summarizer) |
| Summary source text | contentText |
| Citation coverage | Summary is a WeSearch-generated derivative; primary citation is the original publisher URL. |
| Cluster | 4nc-sljH9uga |
| Cluster logic | Grouped by semantic title/content similarity across sources within a rolling window. Same-publisher template collisions are excluded from coverage comparison. |
| Ranking reason | Story pages are not engagement-ranked. Hub feeds use recency, with optional source-diversified chronological ordering (cap consecutive stories per source). No personalized ranking. |
| Publisher visit | Yes — open original |
| Substitutes article? | No — link-out required for full text |
Rights status (four layers)
WeSearch handling by dimension
| Indexing | May the item be indexed (stored, ranked, made findable)? | Allowed |
| Snippet | May a short excerpt of the publisher's text be shown? | Allowed |
| AI summary | May WeSearch generate its own short summary of the article? | Limited |
| Retrieval / RAG | May the content be exposed for third-party retrieval-augmented generation? | Not asserted |
| Model training | May the content be used to train AI models? | Not asserted |
| Commercial reuse | May the content be reused commercially? | Not permitted |
Basis: Derived from the published RSS/Atom feed. Contact: [email protected]. Reviewed: 2026-07-24.
Opening excerpt (first ~120 words) tap to expand
Back to Articles Unlocking asynchronicity in continuous batching Published May 14, 2026 Update on GitHub Upvote 33 +27 Rémi Ouazan Reboul ror Follow Pedro Cuenca pcuenq Follow Aritra Roy Gosthipaty ariG23498 Follow Synchronous batching Creating concurrency What is a CUDA stream? Default and non-default streams Back to Continuous Batching Enforcing synchronization What is a CUDA event? Using events in Continuous Batching Filling the vacuum Race conditions Carry-over The full async loop Does it actually work? Conclusion TL;DR: we explain how to separate CPU and GPU workloads to get a massive performance boost for inference. This is the second post in a series on efficient LLM inference. The first post covered continuous batching from first principles.
…
Excerpt limited to ~120 words for fair-use compliance. The full article is at Huggingface.
Discussion
0 commentsMore from Huggingface
