Build a valid robots.txt file in seconds.
Use * for all crawlers, or a specific name like Googlebot.
A robots.txt file is a plain-text file at the root of your domain that tells crawlers which parts of your site they may fetch. It is the first thing well-behaved bots like Googlebot request, and a single typo in it can accidentally block your whole site from being crawled — which makes generating it with a form safer than hand-editing.
This robots.txt generator builds the file from structured inputs. Create one or more rule groups, each with a User-agent (use * for all crawlers or a specific name like Googlebot), Allow and Disallow paths, and an optional Crawl-delay. You can add multiple sitemap URLs, which are appended as Sitemap lines so crawlers can discover every page you want indexed. The output panel updates live, and invalid sitemap URLs are flagged before they reach the file.
The file is generated entirely in your browser. When it looks right, copy the text or download it as robots.txt and upload it to the root of your site, for example example.com/robots.txt — crawlers will not find it in a subdirectory. Remember that robots.txt controls crawling, not indexing; pages you need kept out of search results entirely should use a noindex directive or authentication instead.
In the first rule group, enter * to target all crawlers or a specific bot name like Googlebot.
List the paths each crawler may or may not fetch, such as Disallow /admin. Use the Add path buttons for additional entries and set an optional Crawl-delay.
Click Add rule group to give different crawlers different rules, for example stricter rules for a specific bot.
Enter the full URL of each sitemap, like https://example.com/sitemap.xml, so crawlers can discover your pages.
Click Copy to grab the text or Download to save robots.txt, then upload it to the root of your domain.
Not reliably. Robots.txt blocks crawling, but a blocked URL can still be indexed without content if other sites link to it. To keep a page out of search results, allow crawling and add a noindex meta tag, or protect the page with authentication.
At the root of the host, for example https://example.com/robots.txt. Crawlers only look there; a robots.txt placed in a subdirectory has no effect, and each subdomain needs its own file.
The asterisk is a wildcard matching every crawler that does not have a more specific group in the file. A bot follows the most specific group that matches its name and ignores the rest.
No, Googlebot ignores the Crawl-delay directive; you control Google crawl rate through Search Console instead. Some other crawlers, such as some SEO and archive bots, do honor it.
Crawlers assume everything is allowed and crawl the whole site. That is fine for many small sites, but a robots.txt is still useful for declaring sitemaps and keeping bots out of admin areas, search result pages, and duplicate content.
Yes. A group containing Disallow with no path permits the crawler to access everything, while Disallow / blocks the entire site, so the single slash makes all the difference.
Custom domains, branded URLs, click analytics. Everything you need to make every link work harder.