Compute Optimal Tokenization: Scaling Laws for Data Compression in LLMs
The article discusses findings on optimal tokenization for data compression in large language models (LLMs). It highlights that the ideal bytes-to-parameter ratio remains consistent across various compression rates and compute budgets. Additionally, the optimal compression rate varies by language, with implications for training models in different linguistic contexts.
- ▪The optimal ratio between bytes of data and model parameters is approximately constant across varying compute budgets and compression rates.
- ▪At each training compute budget, there is an optimal compression rate that minimizes loss, which decreases as the training budget increases.
- ▪The optimal bytes-to-parameter ratio and compression rate vary across different languages, showing correlation with the Parity metric.
Hacker News (Newest) files mainly under programming. We currently carry 5,306 of its stories.
Story provenance
Source · retrieval · rights · ranking — open for full record
inspect →
Story provenance
Attribution is not the same as permission. This drawer separates discovery metadata, excerpts, WeSearch-generated summaries, reuse status, and whether the publisher receives the visit. Nothing here claims a legal grant the publisher has not made.
Record
| Original publisher | Github |
| Canonical URL | https://co-tok.github.io/ |
| Publication time | Tue, 19 May 2026 03:21:20 +0000 |
| Retrieval time | 2026-05-19T03:34:57.258Z |
| Last seen | 2026-05-19T03:34:57.258Z |
| Headline source | Publisher (no WeSearch rewrite) |
| Excerpt source | publisher body |
| Excerpt method | First ~120 words (~800 chars) of extracted publisher body, fair-use limited. |
| Summary | WeSearch · cerebras-chat (WeSearch summarizer) |
| Summary source text | contentText |
| Citation coverage | Summary is a WeSearch-generated derivative; primary citation is the original publisher URL. |
| Cluster | 5dbrMDoBO6gm |
| Cluster logic | Grouped by semantic title/content similarity across sources within a rolling window. Same-publisher template collisions are excluded from coverage comparison. |
| Ranking reason | Story pages are not engagement-ranked. Hub feeds use recency, with optional source-diversified chronological ordering (cap consecutive stories per source). No personalized ranking. |
| Publisher visit | Yes — open original |
| Substitutes article? | No — link-out required for full text |
Rights status (four layers)
WeSearch handling by dimension
| Indexing | May the item be indexed (stored, ranked, made findable)? | Allowed |
| Snippet | May a short excerpt of the publisher's text be shown? | Allowed |
| AI summary | May WeSearch generate its own short summary of the article? | Limited |
| Retrieval / RAG | May the content be exposed for third-party retrieval-augmented generation? | Not asserted |
| Model training | May the content be used to train AI models? | Not asserted |
| Commercial reuse | May the content be reused commercially? | Not permitted |
Basis: Derived from the published RSS/Atom feed. Contact: [email protected]. Reviewed: 2026-07-24.
Opening excerpt (first ~120 words) tap to expand
[F1] Optimal Data to Model Size For a fixed compute budget (1e20 FLOPs), we plot loss against compression rate and bytes per parameter ratio. This yields a 3D IsoFLOP: The bowl-shaped IsoFLOP surface shows that, for every compression rate, the lowest loss is achieved at roughly the same bytes per parameter ratio (triangles). Interactive version of the plot above. You can rotate it and hover over points to check loss, compression, and bytes per parameter values: We observe that the optimal bytes per parameter ratio remains nearly constant across different compression rates. Finding 1 The optimal ratio between bytes of data and model parameters is approximately constant across varying compute budgets and compression rates.
…
Excerpt limited to ~120 words for fair-use compliance. The full article is at Github.
Discussion
0 commentsMore from Github
