Content-defined chunking in Bazel's remote cache
BuildBuddy has implemented Content-Defined Chunking (CDC) in Bazel's remote cache to reduce data transfer and storage by reusing unchanged portions of build outputs. This approach allows only the changed bytes to be uploaded or downloaded, rather than entire files, improving efficiency for large binaries and packages. The feature is available in Bazel 8.7 or 9.1+ using the --experimental_remote_cache_chunking flag.
- ▪BuildBuddy's Remote Cache uses Content-Defined Chunking to reuse chunks of data that have not changed.
- ▪In benchmarks on BuildBuddy's own repository, CDC reduced upload sizes by 40% and disk cache usage by 40%.
- ▪CDC helps optimize transitive actions like linking, bundling, and packaging, where small input changes result in large output files with mostly identical content.
- ▪The CDC feature requires Bazel 8.7 or 9.1+ and must be enabled with the --experimental_remote_cache_chunking flag.
- ▪Without CDC, remote caches treat slightly modified large files as entirely new blobs, leading to inefficient data transfers and storage duplication.
Lobsters files mainly under programming. We currently carry 187 of its stories.
Story provenance
Source · retrieval · rights · ranking — open for full record
inspect →
Story provenance
Attribution is not the same as permission. This drawer separates discovery metadata, excerpts, WeSearch-generated summaries, reuse status, and whether the publisher receives the visit. Nothing here claims a legal grant the publisher has not made.
Record
| Original publisher | Buildbuddy |
| Canonical URL | https://www.buildbuddy.io/blog/content-defined-chunking |
| Publication time | Sat, 16 May 2026 22:29:58 -0500 |
| Retrieval time | 2026-05-17T04:03:58.374Z |
| Last seen | 2026-05-17T04:03:58.374Z |
| Headline source | Publisher (no WeSearch rewrite) |
| Excerpt source | publisher body |
| Excerpt method | First ~120 words (~800 chars) of extracted publisher body, fair-use limited. |
| Summary | WeSearch · cerebras-chat (WeSearch summarizer) |
| Summary source text | contentText |
| Citation coverage | Summary is a WeSearch-generated derivative; primary citation is the original publisher URL. |
| Cluster | kDsD9A_CqSWd |
| Cluster logic | Grouped by semantic title/content similarity across sources within a rolling window. Same-publisher template collisions are excluded from coverage comparison. |
| Ranking reason | Story pages are not engagement-ranked. Hub feeds use recency, with optional source-diversified chronological ordering (cap consecutive stories per source). No personalized ranking. |
| Publisher visit | Yes — open original |
| Substitutes article? | No — link-out required for full text |
Rights status (four layers)
WeSearch handling by dimension
| Indexing | May the item be indexed (stored, ranked, made findable)? | Allowed |
| Snippet | May a short excerpt of the publisher's text be shown? | Allowed |
| AI summary | May WeSearch generate its own short summary of the article? | Limited |
| Retrieval / RAG | May the content be exposed for third-party retrieval-augmented generation? | Not asserted |
| Model training | May the content be used to train AI models? | Not asserted |
| Commercial reuse | May the content be reused commercially? | Not permitted |
Basis: Derived from the published RSS/Atom feed. Contact: [email protected]. Reviewed: 2026-07-24.
Opening excerpt (first ~120 words) tap to expand
Remote Cache CDC: Reusing BytesTyler French, Engineer @ BuildBuddyMay 1, 2026 · 15 min read The goal: move the changed bytes, not the whole output. BuildBuddy's Remote Cache uses Content-Defined Chunking (CDC) to make large build outputs behave more incrementally. When a binary, bundle, package, or archive is mostly unchanged, BuildBuddy can reuse chunks it has already seen instead of re-uploading or re-downloading the entire file. In our Bazel chunking implementation PR, we observed 40% less data uploaded and a 40% smaller disk cache when benchmarked on BuildBuddy's own repo. To enable client-side CDC with BuildBuddy, use Bazel 8.7 or 9.1+ and pass --experimental_remote_cache_chunking. Setting the Scene The next frontier for build caching is not just skipping actions.
…
Excerpt limited to ~120 words for fair-use compliance. The full article is at Buildbuddy.
Discussion
0 commentsMore from Buildbuddy
