Labs

Robots.txt & Sitemap Generator

Build a correct robots.txt with presets and rule groups, and turn a list of URLs into a valid sitemap.xml — both ready to download.

Add a rule group or paste URLs
Builder —
File
Presets
Output —
Your robots.txt or sitemap.xml will appear here
Settings lastmod today · no changefreq · no priority
How It Works

How to Use Robots.txt & Sitemap Generator in 3 Steps

1

Pick a preset or add groups

Start from Allow everything, Block AI crawlers or WordPress, then adjust: each rule group has one or more user-agents (comma-separated), Disallow and Allow paths (one per line) and an optional crawl-delay. Enter your site URL to get the Sitemap: line.

2

Paste URLs for the sitemap

Switch to sitemap.xml and paste one URL per line — relative paths like /about are resolved against your site URL. Add , lastmod, changefreq, priority after a URL to override the defaults for that page.

3

Download both files

Problems (foreign hosts, bad dates, duplicates) are listed under the output. Download robots.txt and sitemap.xml and upload them to the root of your site, then submit the sitemap in Search Console.

FAQ

Robots.txt & Sitemap Generator — Frequently Asked Questions

Does blocking AI crawlers in robots.txt actually work?

Reputable crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot, PerplexityBot, Bytespider and others in the preset) state that they honour robots.txt. It is a request, not a lock — content stays publicly reachable, and bots that ignore the standard are not stopped. The preset list is current as of October 2026; add or remove agents freely.

Why is there an empty Disallow: line?

A group with a user-agent and an empty Disallow is the standard way to say “this agent may crawl everything”. Omitting the line entirely is ambiguous to some crawlers, so the generator always writes it.

Why not just crawl my site to build the sitemap?

Toolity runs entirely in your browser and has no server, so it cannot crawl. Paste the URLs from your CMS, an export, or your existing sitemap. Up to 50,000 URLs per file (the sitemap protocol limit); split larger sites into several files and list them in a sitemap index.

What do lastmod, changefreq and priority do?

lastmod (YYYY-MM-DD) is the one search engines actually use — keep it honest. changefreq and priority are hints that Google ignores; they are omitted by default and only written when you set them.

Do the files need to be at a specific location?

Yes: https://your-site.com/robots.txt — exactly at the root of the host. The sitemap can live anywhere on the same host, but the root is conventional and is what the Sitemap: line points to.

Is my input sent anywhere?

No. Everything is computed by JavaScript in your browser tab. Nothing is uploaded, logged or stored by Toolity.