LLMs Locally With a CPU? I Tested 8 Models on Linux
Recent advancements in model formats and quantization techniques have made it possible to run large language models (LLMs) on CPU-only setups. The key metric for usability is the tokens per second rate, which determines how responsive a model feels during use. This article provides insights into which models are suitable for low-end machines based on practical testing.
- ▪Newer model formats and aggressive quantization have made LLMs smaller and lighter, allowing them to run on CPUs.
- ▪The usability of a model is determined by its tokens per second rate, with 15–30 tok/s being optimal for everyday use.
- ▪Models in the 1B-2B range consistently offer a good balance of performance and resource usage on typical older laptops.
Hacker News (Newest) files mainly under programming. We currently carry 5,306 of its stories.
Story provenance
Source · retrieval · rights · ranking — open for full record
inspect →
Story provenance
Attribution is not the same as permission. This drawer separates discovery metadata, excerpts, WeSearch-generated summaries, reuse status, and whether the publisher receives the visit. Nothing here claims a legal grant the publisher has not made.
Record
| Original publisher | It's FOSS |
| Canonical URL | https://itsfoss.com/testing-local-llms-without-gpu/ |
| Publication time | Mon, 25 May 2026 08:05:52 +0000 |
| Retrieval time | 2026-05-25T08:37:36.531Z |
| Last seen | 2026-05-25T08:37:36.531Z |
| Headline source | Publisher (no WeSearch rewrite) |
| Excerpt source | publisher body |
| Excerpt method | First ~120 words (~800 chars) of extracted publisher body, fair-use limited. |
| Summary | WeSearch · cerebras-chat (WeSearch summarizer) |
| Summary source text | contentText |
| Citation coverage | Summary is a WeSearch-generated derivative; primary citation is the original publisher URL. |
| Cluster | KoBem07A-YHb |
| Cluster logic | Grouped by semantic title/content similarity across sources within a rolling window. Same-publisher template collisions are excluded from coverage comparison. |
| Ranking reason | Story pages are not engagement-ranked. Hub feeds use recency, with optional source-diversified chronological ordering (cap consecutive stories per source). No personalized ranking. |
| Publisher visit | Yes — open original |
| Substitutes article? | No — link-out required for full text |
Rights status (four layers)
WeSearch handling by dimension
| Indexing | May the item be indexed (stored, ranked, made findable)? | Allowed |
| Snippet | May a short excerpt of the publisher's text be shown? | Allowed |
| AI summary | May WeSearch generate its own short summary of the article? | Limited |
| Retrieval / RAG | May the content be exposed for third-party retrieval-augmented generation? | Not asserted |
| Model training | May the content be used to train AI models? | Not asserted |
| Commercial reuse | May the content be reused commercially? | Not permitted |
Basis: Derived from the published RSS/Atom feed. Contact: [email protected]. Reviewed: 2026-07-24.
Opening excerpt (first ~120 words) tap to expand
For the longest time, I assumed running LLMs locally needed a decent GPU. That’s what most guides implied, and honestly, that’s how the ecosystem felt not too long ago. But after digging into recent tools and actually trying things out on CPU-only setups, that assumption doesn’t really hold anymore.Newer model formats like GGUF and aggressive quantization (think 4-bit variants) have made these models much smaller and lighter. At the same time, runtimes such as Llama.cpp have become efficient enough that CPUs (yes, even older ones) can run them without completely falling apart.That said, I quickly realized something more important: just because a model runs doesn’t mean it’s usable.While testing, I found that the real metric that matters isn’t model size or even RAM usage, it’s actually…
Excerpt limited to ~120 words for fair-use compliance. The full article is at It's FOSS.
Discussion
0 commentsMore from It's FOSS
