Marketing · 8 min read · Updated September 21, 2026

Robots.txt and Sitemap.xml: What to Block, What to List

Two plain-text files sit at the root of almost every indexable site, and both are widely misused in the same direction: people treat robots.txt as a way to hide a page from Google, and treat sitemap.xml as a way to make Google rank one higher. Neither is true. Robots.txt controls crawling, sitemap.xml assists discovery, and mixing up "crawl" with "index" is exactly what produces a page that is blocked from crawling yet still shows up in search results with no snippet underneath it.

This guide covers what each file actually does, the directives worth using, what belongs in a sitemap and what quietly ruins one, and where short links and redirects fit into both, since a site built around them runs into decisions a static brochure site never has to make.

Two files, two different jobs

A robots.txt at the root of a domain tells well-behaved crawlers which paths they may request. It is a request, not an enforcement mechanism: it works because Googlebot, Bingbot and other major crawlers choose to honor it, not because anything blocks the request at the network level. A scraper that ignores robots.txt simply fetches the page anyway.

A sitemap.xml at the root (or referenced from robots.txt) lists URLs you consider worth crawling, optionally with a last-modified date. It is a hint that speeds up discovery, particularly for pages with few internal links pointing at them. It does not force a page into the index, does not influence ranking, and a crawler is free to ignore individual entries or the file entirely.

What robots.txt directives actually do

The format is small enough to hand-write, which is exactly how it accumulates mistakes: a trailing slash left off a Disallow line, a User-agent block that silently overrides a more specific one above it, a Crawl-delay nobody respects.

Rules are matched by the most specific path, not by order, and an empty Disallow: value means "block nothing" for that block, which is a common way to accidentally allow everything. Build the file with a robots.txt generator rather than hand-editing it if you are not confident on those two points; the form only lets you produce syntax that parses correctly.

A worked example: what ReSlug blocks and why

Every entry in a good robots.txt should have a reason you could state out loud. Here is the reasoning behind a real one, ReSlug’s own:

User-agent: *
Allow: /
Disallow: /app/
Disallow: /api/
Disallow: /r/
Disallow: /login
Disallow: /signup
Disallow: /auth/

Sitemap: https://reslug.com/sitemap.xml

• `/app/` is the signed-in dashboard. It requires a login, has no content a logged-out crawler could render, and every URL under it is either private or a duplicate shell for every account. Nothing there should ever appear in search results.
• `/api/` is machine-to-machine JSON, never meant to render as a page. Blocking it is mostly hygiene: no crawler would index a JSON response usefully, but there is no reason to spend crawl budget confirming that.
• `/r/` is the short-link redirect namespace, and this is the one worth pausing on. A redirect endpoint has no content of its own to index, either the destination is what should rank, or nothing should. Letting a crawler spend requests hopping through redirect URLs burns crawl budget on pages that will never earn a snippet, and on a high-volume shortener that budget adds up fast.
• Auth pages (/login, /signup, password reset and similar) carry no unique content, appear identically for every visitor, and would otherwise compete with the marketing pages that are actually meant to rank for related terms.

Notice what is not on that list: /tools/, /guides/, /pricing, and every other marketing page are covered by the blanket Allow: / and never mentioned, because the default for anything not explicitly disallowed is already "crawl it." A robots.txt only needs to name the exceptions.

Blocking a crawler is not the same as blocking the index

This is the mistake worth fixing first, because it runs backwards from what most people assume. If a page must never appear in search results at all, even without a snippet, Disallow is the wrong tool, because a disallowed page can still be indexed by URL alone if something links to it. The correct tool is a noindex directive, either a <meta name="robots" content="noindex"> tag in the page, or an X-Robots-Tag: noindex response header for pages where a meta tag is not possible (a PDF, an XML file, a non-HTML API response).

The catch is that noindex only works if the crawler is allowed to fetch the page and read the tag. A URL that is both disallowed and noindexed is a contradiction: the crawler never gets far enough to see the instruction telling it to stay out of the index, so the disallow (which does not guarantee removal) is the only directive actually in effect. Use noindex on pages that must be readable by crawlers but excluded from results; use disallow on paths that should never be requested at all.

ReSlug applies this distinction at the infrastructure level rather than per page: every response from a non-production host (a preview deploy, a staging domain) carries an X-Robots-Tag: noindex, nofollow header, set at the edge before the request reaches the app. A header covers every response type, including the sitemap and static assets, and it cannot be accidentally disabled by a frontend bug the way a missing meta tag could. Production is the only host exempt from that header, which keeps every non-production copy of the site out of search results by default rather than by remembering to opt in.

What belongs in a sitemap

A sitemap should list exactly the canonical, indexable URLs you want discovered, nothing more. Padding it with disallowed paths, redirects, or parameterized duplicates does not help; it teaches the crawler that a fraction of your sitemap is noise, which makes it trust the whole file less.

The lastmod field is the one field worth getting right, and the trap is subtle: stamping every URL with the current build date on every deploy is worse than leaving it out. A crawler that sees all 40-odd URLs "change" on the same day, every day, learns to stop trusting the field and falls back to its own schedule. ReSlug’s sitemap generator instead sets lastmod from the last git commit date of the file that actually renders each page, so a URL’s lastmod only moves on a deploy that changed that specific page, and changefreq/priority are left out entirely rather than filled with a guess Google discards anyway.

What does not belong in a sitemap

Two categories consistently sneak into shortener and bio-page sitemaps that should not be there.

Redirect URLs, first. A short link is a 302 to somewhere else (see 301 vs 302 redirects for why it should stay a 302), and a sitemap entry for it would list a URL whose entire content is "go elsewhere." If the destination is worth indexing, list the destination on its own site’s sitemap; the redirect adds a hop for the crawler with no additional content to show for it.

User-generated pages that change on their own schedule, second, such as a bio-link page created and edited by an individual account. These are not absent from search by accident: they change whenever the owner edits them, there can be thousands created per day, and forcing them through a build-time sitemap would mean either a stale list or a sitemap that regenerates continuously for content the operator does not control. ReSlug’s own build-time sitemap only lists the fixed marketing, tool and guide routes for exactly this reason; individual bio pages are excluded on purpose.

That does not mean those pages are invisible to bots generally, only that a sitemap is the wrong mechanism for them. Preview scrapers (Slackbot, Facebook’s crawler) still need to read a bio page’s title and image to build a link card when someone shares it, which is a request-time problem, not a discovery-time one: the server recognizes the bot user agent and returns tags for that specific page on demand, the same pattern covered in link previews and short links. Search discoverability and share-card correctness are different jobs, solved differently, even though both involve a bot reading a page.

Where the files go, and how to check them

Both files must live at the domain root: https://example.com/robots.txt and https://example.com/sitemap.xml. A robots.txt at /blog/robots.txt or a sitemap referenced only from a page nobody links to is invisible to crawlers, they check the root by convention and nowhere else unless you tell them to via the Sitemap: line.

If you run short links or a bio page on a custom domain separate from your main site, that domain has its own root and needs its own robots.txt if you want anything other than the crawler default of "everything is allowed." A robots.txt on your main domain has no effect on a subdomain or a different domain entirely.

1. Fetch each file directly and confirm it returns 200 with the content type you expect: curl -I https://example.com/robots.txt and the same for /sitemap.xml.
2. Read the robots.txt by eye for the mistake that matters most: an accidental Disallow: / left over from a staging config, which blocks the entire site.
3. Check that every Sitemap: line in robots.txt points to a URL that actually resolves, and that every loc in the sitemap is a canonical URL you would want to rank, not a redirect or a parameterized duplicate.
4. In Google Search Console, use the Sitemaps report to confirm the file was fetched successfully, and the URL Inspection tool on a specific page to see whether Google considers it indexed, crawled but not indexed, or blocked, three different states that a robots.txt reading alone cannot tell you apart.

A sitemap generator and robots.txt generator turn a form into valid syntax for both files without hand-writing XML or memorizing directive names, which is most of the value: the failure mode for both files is almost always a syntax slip, not a strategic one.