How to Write a Robots.txt File for Your Small Business Website

May 17, 2026 · 10:00 AM UTC ·7 min read · 0 reactions · 0 comments · 32 views

#web development #seo #small business #robots.txt #search engines

TL;DR · WeSearch summary

A robots.txt file provides instructions to search engine crawlers about which parts of a website they can or cannot access. It is not a security tool but helps guide well-behaved search engines like Google and Bing during indexing. For most small business websites, a simple robots.txt file allowing full access is sufficient and helps ensure proper visibility in search results.

Key facts

▪A robots.txt file is placed at the root of a website and tells search engine robots which pages they can or cannot crawl.
▪The directive 'User-agent: *' applies rules to all search engine bots, and 'Disallow:' with no path allows full access to the site.
▪Blocking important files like CSS or JavaScript in robots.txt can harm search engine indexing and should be avoided.
▪A misconfigured robots.txt file, such as one with 'Disallow: /', can prevent a website from appearing in search results.
▪Robots.txt should never be used to hide sensitive information, as it is not a security measure and can be ignored by malicious bots.

How this story was covered

2 outlets in our directory ran this story, first to last over 9 hours. All of the coverage we found sits in one bucket: centre. That one-sidedness is itself worth noticing.

Centre · 1

▪ How are small businesses handling website/app development costs in 2026? — r/smallbusiness

About this source

DEV.to (Top) files mainly under programming. We currently carry 4,924 of its stories.

All DEV.to (Top) coverage →

Original article

DEV.to (Top)

Read full at DEV.to (Top) →

Story provenance

Source · retrieval · rights · ranking — open for full record

inspect →

Attribution is not the same as permission. This drawer separates discovery metadata, excerpts, WeSearch-generated summaries, reuse status, and whether the publisher receives the visit. Nothing here claims a legal grant the publisher has not made.

Record

Original publisher	DEV.to (Top)
Canonical URL	https://dev.to/auditmyadmin/how-to-write-a-robotstxt-file-for-your-small-business-website-5b4k
Publication time	Sun, 17 May 2026 10:00:22 +0000
Retrieval time	2026-05-17T10:22:13.044Z
Last seen	2026-05-17T10:22:13.044Z
Headline source	Publisher (no WeSearch rewrite)
Excerpt source	publisher body
Excerpt method	First ~120 words (~800 chars) of extracted publisher body, fair-use limited.
Summary	WeSearch · cerebras-chat (WeSearch summarizer)
Summary source text	contentText
Citation coverage	Summary is a WeSearch-generated derivative; primary citation is the original publisher URL.
Cluster	HDJpd_FznNvK · 2 stories
Cluster logic	Grouped by semantic title/content similarity across sources within a rolling window. Same-publisher template collisions are excluded from coverage comparison.
Ranking reason	Story pages are not engagement-ranked. Hub feeds use recency, with optional source-diversified chronological ordering (cap consecutive stories per source). No personalized ranking.
Publisher visit	Yes — open original
Substitutes article?	No — link-out required for full text

Rights status (four layers)

Publisher-declared

No publisher-confirmed rights record for this source yet.

Machine-readable

No source-specific machine-readable restriction detected beyond the public feed.

WeSearch interpretation

WeSearch declared handling (basis: Derived from the published RSS/Atom feed). This is WeSearch policy, not a legal grant on the publisher's behalf.

Unknown

Retrieval and training permissions are not asserted unless the publisher confirms them.

WeSearch handling by dimension

Indexing	May the item be indexed (stored, ranked, made findable)?	Allowed
Snippet	May a short excerpt of the publisher's text be shown?	Allowed
AI summary	May WeSearch generate its own short summary of the article?	Limited
Retrieval / RAG	May the content be exposed for third-party retrieval-augmented generation?	Not asserted
Model training	May the content be used to train AI models?	Not asserted
Commercial reuse	May the content be reused commercially?	Not permitted

Basis: Derived from the published RSS/Atom feed. Contact: [email protected]. Reviewed: 2026-07-24.

Open original publisher → Source rights JSON → Machine-readable provenance → Rights framework → WESEARCH-STORY-PROVENANCE-V1

Opening excerpt (first ~120 words) tap to expand

try { if(localStorage) { let currentUser = localStorage.getItem('current_user'); if (currentUser) { currentUser = JSON.parse(currentUser); if (currentUser.id === 3910804) { document.getElementById('article-show-container').classList.add('current-user-is-article-author'); } } } } catch (e) { console.error(e); } Daniel Fenton Posted on May 17 • Originally published at website.auditmy.co.uk How to Write a Robots.txt File for Your Small Business Website #beginners #tutorial #web #webdev What on earth is a robots.txt file? Imagine you run a bakery and a journalist turns up wanting to photograph everything. You might say, "Yes, photograph the front of house, but please don't go into the kitchen or the staff room." A robots.txt file does exactly that for your website.

…

Excerpt limited to ~120 words for fair-use compliance. The full article is at DEV.to (Top).

Anonymous · no account needed

Discussion

0 comments