Your robots.txt file tells crawlers which parts of your site they are welcome to read. Pick a starting point, tick what applies, and copy the result.
Save as robots.txt in your site root, so it answers at example.com/robots.txt. On WordPress, Rank Math and Yoast both have a box for this that avoids creating a real file.
The thing to understand before anything else
robots.txt is a request, not a lock.
Google and Bing respect it. Plenty of crawlers do not, and nothing forces them to. Anyone can read your robots.txt by visiting yoursite.com/robots.txt, which means listing a secret folder there is an advertisement, not protection.
If something genuinely must not be reached, use a password or a noindex tag. Use robots.txt for tidying up what search engines spend their time on, which is what it is actually for.
Blocked is not the same as hidden
A page blocked in robots.txt can still appear in Google. It shows up with no description and a note saying no information is available, because Google knows the page exists from links pointing at it but was not allowed to read it.
Worse, blocking a page in robots.txt makes it impossible for Google to see a noindex tag on that page, because it cannot read the page to find the tag. So if you want something genuinely out of search, use noindex and leave robots.txt alone. Doing both keeps it in.
That trips up a lot of people, and it is the reverse of what most guides imply.
About the AI crawler options
This is the part most generators have not caught up with, and the distinction matters more than the tick box makes it look.
Training crawlers
GPTBot, ClaudeBot, CCBot, Google-Extended, Bytespider and the rest collect pages to train models. They read your content and send you nothing back. Blocking them costs you no traffic whatsoever.
Whether you want to is a judgement call, not a technical one. Some people object to their work being used this way. Others take the view that appearing in an AI’s knowledge is worth something eventually. Both positions are reasonable.
Search assistants
OAI-SearchBot, ChatGPT-User and Perplexity-User are different. They fetch a page because someone asked a question right now, and they cite the source. That citation is a link, and people click it.
Blocking those removes you from a genuinely growing source of visitors, at exactly the point when it is growing.
Almost every “block AI bots” list you will find online mixes the two groups together, which is why the tool keeps the assistants allowed by default even when you block training. If you want both gone, untick the second box, but do it knowing what it costs.
What the WordPress defaults do
Blocking wp-admin, allowing admin-ajax.php
The admin area has nothing worth indexing, so keeping crawlers out saves them wasting time there.
The second line matters and is easy to miss. Your public pages use admin-ajax.php for ordinary things like forms, filters and carts. Block the folder without that exception and Google reports blocked resources when it renders your pages, which can affect how it judges them.
Leave both ticked together. The tool warns you if you untick the second one.
Blocking readme.html and license.txt
Those two files sit in every WordPress install and readme.html announces your exact version number. It is a small thing, but there is no reason to leave it discoverable.
Being honest: this hides them from crawlers, not from a person who types the address. To actually block them, use the Hide sensitive files option in our htaccess generator.
Blocking search result pages
Off by default because it depends on your site. Your internal search creates a new address for every query anyone ever types, and those pages are thin and endless. On a site with a lot of search traffic, blocking them stops Google wasting its time. On a small site it makes little difference.
The sitemap line
The only line in the whole file that actively helps rather than restricts. It tells crawlers where your list of pages is.
With Rank Math or Yoast it is usually https://yoursite.com/sitemap_index.xml. Open that address in a browser first to check it loads before pasting it in.
Where the file goes
It has to answer at yoursite.com/robots.txt exactly. Not in a subfolder, not with a different name. Crawlers look in that one place and nowhere else.
On WordPress, use your SEO plugin
WordPress serves a robots.txt of its own even when no file exists. Both Rank Math and Yoast give you a box to edit it, and that is the tidiest route.
- Rank Math: General Settings, then Edit robots.txt.
- Yoast: Tools, then File editor.
One catch worth knowing: if a real robots.txt file exists on the server, it wins and the plugin box stops having any effect. That is the usual reason people edit the setting, save it, and see no change. Check your file manager for an actual file first.
Or upload a file
Create robots.txt in public_html, paste, save. Then visit the address in a browser and confirm you see what you expect.
Checking it worked
- Visit
yoursite.com/robots.txtin a private window. You should see plain text. - Open Search Console and use the robots.txt report, which shows what Google last fetched and flags any lines it could not understand.
- Check a specific page with the URL Inspection tool if you are unsure whether it is blocked.
Google caches the file for around a day, so changes are not instant.
Mistakes worth avoiding
Leaving Disallow: / on a live site
The classic. A site is built on staging with everything blocked, then moved to the live domain with the file intact. Traffic never arrives and nobody thinks to look, because the site itself works perfectly.
If a site launched recently and search traffic is simply absent, check this first. It takes ten seconds and it is the answer more often than it should be.
Blocking CSS and JavaScript
Old guides suggested blocking /wp-includes/ and the plugins folder. Do not. Google renders your pages like a browser, and blocking those files means it sees a broken layout and judges the page on that.
Expecting it to hide anything
Worth saying twice. Your robots.txt is public. Listing /secret-client-files/ in it tells everyone the folder exists and exactly what it is called.
Questions people ask
Do I even need one?
No. A site with no robots.txt is treated as fully open, which is fine for most small sites. Where it earns its place is the sitemap line and, now, the AI crawler choice.
Will blocking a page remove it from Google?
No, and it can make removal harder, for the reason explained near the top. To remove a page, use noindex and let Google read it.
Does blocking bots make my site faster?
A little, on a small server with a lot of crawler traffic. But it only stops the well behaved ones, and those are rarely the problem. For the badly behaved, you need the htaccess route, which refuses them at the server rather than asking nicely.
Is the file case sensitive?
The filename must be lowercase. The paths inside are case sensitive too, so /Private/ and /private/ are different things.
Can I have different rules for different bots?
Yes, which is exactly what the AI and SEO sections produce. Each bot reads only the block addressed to it by name, and falls back to the * block if there is not one. Note that a bot with its own block ignores the * rules entirely, which surprises people.