Implications of Predicting the Next Token
The article discusses the common misconception surrounding the concept of predicting the next token in language models. It contrasts the capabilities of Markov chains with those of more advanced language models, emphasizing that the latter can produce more nuanced and coherent text. The author illustrates the limitations of Markov chains and highlights the importance of understanding the sophistication of modern language generation techniques.
- ▪Many people struggle to grasp what it means to predict the next token in language models.
- ▪Markov chains are incapable of producing meaningful text, often resulting in gibberish.
- ▪Advanced language models achieve a level of sophistication that far surpasses the outputs of Markov chains.
Hacker News (Newest) files mainly under programming. We currently carry 5,306 of its stories.
Story provenance
Source · retrieval · rights · ranking — open for full record
inspect →
Story provenance
Attribution is not the same as permission. This drawer separates discovery metadata, excerpts, WeSearch-generated summaries, reuse status, and whether the publisher receives the visit. Nothing here claims a legal grant the publisher has not made.
Record
| Original publisher | Lesswrong |
| Canonical URL | https://www.lesswrong.com/posts/AzRRPDNmeEoJdSiib/implications-of-predicting-the-next-token |
| Publication time | Mon, 25 May 2026 15:47:04 +0000 |
| Retrieval time | 2026-05-25T16:07:38.273Z |
| Last seen | 2026-05-25T16:07:38.273Z |
| Headline source | Publisher (no WeSearch rewrite) |
| Excerpt source | publisher body |
| Excerpt method | First ~120 words (~800 chars) of extracted publisher body, fair-use limited. |
| Summary | WeSearch · cerebras-chat (WeSearch summarizer) |
| Summary source text | contentText |
| Citation coverage | Summary is a WeSearch-generated derivative; primary citation is the original publisher URL. |
| Cluster | Dap8QVh--XU2 |
| Cluster logic | Grouped by semantic title/content similarity across sources within a rolling window. Same-publisher template collisions are excluded from coverage comparison. |
| Ranking reason | Story pages are not engagement-ranked. Hub feeds use recency, with optional source-diversified chronological ordering (cap consecutive stories per source). No personalized ranking. |
| Publisher visit | Yes — open original |
| Substitutes article? | No — link-out required for full text |
Rights status (four layers)
WeSearch handling by dimension
| Indexing | May the item be indexed (stored, ranked, made findable)? | Allowed |
| Snippet | May a short excerpt of the publisher's text be shown? | Allowed |
| AI summary | May WeSearch generate its own short summary of the article? | Limited |
| Retrieval / RAG | May the content be exposed for third-party retrieval-augmented generation? | Not asserted |
| Model training | May the content be used to train AI models? | Not asserted |
| Commercial reuse | May the content be reused commercially? | Not permitted |
Basis: Derived from the published RSS/Atom feed. Contact: [email protected]. Reviewed: 2026-07-24.
Opening excerpt (first ~120 words) tap to expand
I find that a lot of people have trouble with this concept of predicting the next token. And by trouble, I mean that they struggle to understand what it actually means to predict the next token. It seems simpler than it is. Because when you say "predict the next token," I think what most people think of is the Markov chain intuition that you have a big table of statistics, and then you look at what word is the next most likely, and then you pick that as the word. The thing about this is that if you have ever used a Markov chain, you would know that Markov chain text is complete gibberish. Markov chain text does not resemble meaningful writing.
…
Excerpt limited to ~120 words for fair-use compliance. The full article is at Lesswrong.
Discussion
0 commentsMore from Lesswrong
