robots.txt basics for service businesses

The short answer: robots.txt is a public file that gives instructions to crawlers — "do not fetch these paths" — and it does three things people regularly mistake it for doing: it is not access control (anyone can fetch the file and ignore it), it is not a reliable way to remove a URL from search results, and it only works on crawlers that choose to comply (Google: Introduction to robots.txt). For a service business, the file matters because it is where a well-meaning technical edit can silently hide your whole site — and where a private page is often mistakenly relied on for privacy.

How the file works

robots.txt lives at the root of the host: https://example.com/robots.txt. Its rules are grouped by crawler user-agent: a User-agent: line names the crawler the following Allow: and Disallow: lines apply to, with paths matched from the root of the site (Google: Introduction to robots.txt). The formal specification — how compliant crawlers must parse it, longest-match precedence, and the fact that the rules apply per host, protocol and port — is published as RFC 9309, the Robots Exclusion Protocol (IETF RFC 9309).

In plain language: User-agent: * addresses all compliant crawlers; Disallow: / blocks the whole site from those crawlers; Disallow: /private/ blocks one directory. Google documents its own interpretation in detail, including how it treats rule length, wildcards and precedence (Google: How Google interprets robots.txt).

What it is not

Three misconceptions account for most damage:

It is not privacy. The file is public, and non-compliant clients can ignore it entirely. If a page must stay private, use authentication — a login, or removing it from the web entirely. A robots.txt line is a request, not a lock.
It is not a removal tool. A blocked URL can still appear in search results without a snippet. If a page must be out of results, noindex (which requires the crawler to fetch the page — so it must not be blocked in robots.txt) or removal tools are the correct mechanisms (Google: Introduction to robots.txt).
It is not uniform across vendors. Compliance is voluntary per the RFC (IETF RFC 9309), and individual vendors — including the growing number of AI-product crawlers — publish their own guidance about which bots they run and how those bots treat robots.txt. Check each vendor's published crawler documentation before assuming a rule covers it.

Making changes safely

Because one typo can hide a site from search overnight:

Write the intended rules first, on paper, and test them against the paths you care about, using Google's documented parsing and precedence rules as the reference (Google: How Google interprets robots.txt).
Inspect the live file before and after: fetch https://yourdomain/robots.txt directly to see what crawlers actually receive.
Change it in one step, then verify: re-fetch the file and spot-check key pages remain reachable.
Route the edit to the technical owner of the site. robots.txt changes are a configuration change with real consequences; the person who owns the server or CMS settings should make it — this article is orientation, not a runbook.

Why a service business should care

Two practical situations come up often. First, staging or duplicate environments accidentally left crawlable, or worse, the production site accidentally blocked by a copied-over file. Second, deciding which AI-product crawlers to allow or disallow — an access and reach decision that belongs to the site owner, informed by each vendor's published crawler documentation rather than folklore. In both cases the honest framing is the same: robots.txt influences crawl behaviour of compliant crawlers; it does not guarantee inclusion, exclusion, or privacy (IETF RFC 9309). If you want a scoped review of how your site treats crawlers — findings explained, changes you approve, retested under the same conditions — our GEO & SEO services work that way. Write to us via contact.

robots.txt instructs compliant crawlers; it is not a lock.

Frequently asked questions

Can robots.txt keep a page private?

No. The file is public and compliance is voluntary; any client can ignore it (IETF RFC 9309). For privacy, use authentication or take the content off the web.

If I disallow a URL, will it disappear from search results?

Not reliably. A blocked URL can still be indexed and shown without a snippet; for removal, use noindex or the platform's removal tools — and note noindex requires the page not to be blocked in robots.txt (Google: Introduction to robots.txt).

Do AI-product crawlers respect robots.txt?

It varies by vendor. The protocol is voluntary (IETF RFC 9309), and each vendor publishes guidance for its own crawlers. Read the specific vendor's crawler documentation before deciding what your rules achieve.

Who should edit robots.txt on my site?

The technical owner of the site — the person accountable for its server or CMS configuration. Changes are low-effort but high-consequence; anyone advising on content or visibility should propose rules, not apply them unilaterally.

Sources

developers.google.com/search/docs/crawling-indexing/robots/intro (read 10 November 2026)
rfc-editor.org/rfc/rfc9309 (read 10 November 2026)
developers.google.com/search/docs/crawling-indexing/robots/robots_txt (read 10 November 2026)

GEO & SEO optimisation service · Make an enquiry · More GEO and AI search answers

Owned editorial content by TopGEOClaim (ACN International Pte. Ltd., Singapore). Drafted with AI assistance, reviewed, and checked against the linked sources. It describes public documentation, not customer results, and does not promise any search or AI placement. Vendor behaviour may change; check the linked sources.