How to create your robots.txt
Choose whether crawlers are allowed by default, list any folders or pages you want to keep crawlers out of, and add your sitemap address. The file updates as you type. Click Download to save it as robots.txt, then upload it to the root folder of your website so it opens at https://yourdomain.com/robots.txt. In cPanel, that's the public_html folder. A robots.txt file in any other folder is ignored.
What robots.txt does, and what it doesn't
Robots.txt tells well-behaved crawlers which parts of a site they may request. It's used to save crawl effort on pages with no search value, such as carts, internal search results and admin areas. Two things are often misunderstood:
- It doesn't remove pages from Google. A blocked page can still be indexed if other sites link to it, just without a description. To keep a page out of search results, let it be crawled and add a
noindexrobots meta tag instead. - It isn't security. The file is public and bad bots ignore it. Protect private areas with a password, not robots.txt.
Rules explained
User-agent names the crawler a group of rules applies to; * means all crawlers. Disallow blocks a path and everything under it, so /cart/ also blocks /cart/item-1. Allow makes an exception inside a blocked folder. Paths are case-sensitive and must start with a slash. You can use * as a wildcard and $ to mark the end of a URL, for example /*.pdf$ to block PDF files. Sitemap lines point crawlers to your XML sitemap and must use the full address.
Crawl-delay asks a crawler to wait between requests. Bing and Yandex follow it, but Google ignores it; Google adjusts its crawl speed automatically based on how fast your server responds.
Blocking AI crawlers
Tick Block AI training crawlers to stop the main AI companies' crawlers from collecting your content for training: GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Google's AI models), CCBot (Common Crawl), Applebot-Extended, Meta's crawler, Bytespider and PerplexityBot. This doesn't affect normal Google or Bing search rankings. Respecting robots.txt is voluntary, but the major AI companies say their crawlers follow it.
Common mistakes
Never block your CSS or JavaScript files: Google needs them to see your pages the way visitors do. Be careful with Disallow: /, which blocks your whole site and is the most common reason a new site never appears in Google. After uploading, test a few important URLs with Google Search Console's robots.txt report, and check pages with the HTTP Status Code Checker.
Frequently asked questions
Where do I upload robots.txt?
In the root folder of your domain, so it opens at https://yourdomain.com/robots.txt. On cPanel hosting that's the public_html folder.
Does robots.txt stop a page from appearing in Google?
Not reliably. A blocked page can still be indexed from links. Use a noindex meta tag on a crawlable page to keep it out of search results.
Will blocking AI bots hurt my SEO?
No. The AI crawlers blocked here are separate from Googlebot and Bingbot, so normal search rankings aren't affected.
Does Google follow Crawl-delay?
No. Google ignores Crawl-delay. Bing and Yandex respect it.
What does an empty Disallow line mean?
"Disallow:" with nothing after it allows everything. It's the standard way to say all pages may be crawled.
Last updated