Free tool

robots.txt Generator

Build a robots.txt file for your website with a visual editor. Control which crawlers can access your pages, add sitemap references, and download the file. Free, no signup required.

Quick add:

Not supported by Google. Respected by Bing, Yandex, and others.

User-agent: *
Disallow: /admin
Disallow: /api

What Is a robots.txt File?

A robots.txt file is a plain text file placed at the root of your website (e.g., https://example.com/robots.txt) that instructs web crawlers which pages they are allowed to visit. It follows the Robots Exclusion Protocol, a standard that has been in use since 1994. Search engines like Google, Bing, and others check this file before crawling your site. While it is not a security mechanism (it relies on voluntary compliance), it is the primary way to manage crawler traffic and keep private or low-value pages out of search engine indexes.

robots.txt Syntax Guide

A robots.txt file uses a simple line-based syntax. Each directive applies to the user-agent declared above it. Here are the five most common directives.

User-agent

Specifies which crawler the following rules apply to. Use * to target all crawlers, or a specific name like Googlebot or GPTBot.

Allow

Permits crawling of a specific path. Useful for allowing a subdirectory when a parent directory is disallowed. For example, Allow: /blog lets crawlers access your blog even if the rest of the site is blocked.

Disallow

Blocks crawling of a specific path. Disallow: / blocks the entire site. Disallow: /admin blocks only the /admin directory and its contents.

Sitemap

Points crawlers to your XML sitemap. This helps search engines discover all your pages. Use the full URL: Sitemap: https://example.com/sitemap.xml.

Crawl-delay

Sets a delay (in seconds) between successive requests from a crawler. Not supported by Google, but respected by Bing, Yandex, and some other crawlers. Useful for reducing server load.

Common robots.txt Mistakes

  • Blocking your entire site by accident. A Disallow: / rule under User-agent: * prevents all crawlers from accessing any page. This is common on staging sites that get copied to production.
  • Blocking CSS, JS, or image files. If crawlers cannot access your stylesheets and scripts, they cannot render your pages properly. Google specifically recommends against blocking these resources.
  • Using robots.txt for security. The file is publicly accessible and only a suggestion. It does not prevent access; it politely asks crawlers to stay away. Never rely on it to hide sensitive content.
  • Forgetting the trailing slash on directories. Disallow: /admin blocks /admin, /admin/, and /admin/settings. But Disallow: /admin/ only blocks paths under /admin/, not the /admin page itself. Be intentional about which form you use.
  • Not testing after deploying. Always verify your robots.txt is accessible at your domain root and contains the expected rules. Google Search Console has a robots.txt tester you can use.

How to Block AI Crawlers

AI companies use web crawlers to collect training data. If you want to prevent your content from being used to train AI models, add specific rules for each crawler. Here are the most common AI user agents to block.

GPTBot

OpenAI's crawler used to collect data for training GPT models. Blocking this prevents your content from being included in future training datasets.

ChatGPT-User

Used when ChatGPT browses the web in real time to answer user queries. Blocking this prevents your pages from appearing in ChatGPT responses with web browsing enabled.

Claude-Web

Anthropic's web crawler. Blocking it prevents your content from being used by Anthropic's Claude models.

CCBot

Common Crawl's bot that builds open web datasets widely used by AI researchers. Many AI training datasets originate from Common Crawl archives.

Frequently Asked Questions

More Free Tools

QR code generator

Generate custom QR codes for URLs, text, emails, phone numbers, and WiFi networks.

vCard generator

Create professional digital contact cards (VCF files) to easily share your contact information.

CSS gradient generator

Build beautiful CSS gradients with a visual editor. Linear, radial, and conic with live preview.

Box shadow generator

Design CSS box shadows visually with multiple layers, presets, and live preview.

Meta description checker

Check meta title and description length with a live Google SERP preview.

Favicon generator

Upload an image and generate favicons in all standard sizes. Free, instant, no signup.

Twitter/X Card Validator

Preview how your content will look when shared on Twitter/X. Validate meta tags instantly.

Slug generator

Convert any text into clean, SEO-friendly URL slugs. Supports bulk conversion and options.

JSON formatter

Format, validate, and minify JSON with syntax highlighting and tree view.

Base64 encoder/decoder

Encode text to Base64 or decode Base64 to text with full UTF-8 support.

Link-in-bio checklist

Interactive 20-item checklist to optimize your link-in-bio page with progress tracking.

Business name generator

Generate unique, brandable business name ideas from your keywords and style preferences.

CTA generator

Generate high-converting call-to-action button text and supporting microcopy.

Headline analyzer

Score your headlines 0 to 100 and get tips to improve click-through rates.

Invoice generator

Create professional invoices for free. Add line items, tax, and notes, then print or save as PDF.

Testimonial formatter

Format customer testimonials into professional cards. 4 styles, copy as HTML or plain text.

Pricing table generator

Build responsive pricing tables for your website. 3 styles, feature lists, and CTA buttons.

Font generator

Convert text into fancy Unicode fonts for Instagram, TikTok, Twitter, and other social media bios.

Character counter

Count characters, words, and sentences with real-time social media platform limits.

Hashtag checker

Check if your Instagram hashtags are banned or restricted before posting.

OG tag generator

Generate Open Graph and Twitter Card meta tags with live social media previews.

UTM builder

Build UTM-tagged campaign URLs for Google Analytics tracking.

Image size guide

Every social media image dimension in one place, always up to date.

Engagement rate calculator

Calculate your social media engagement rate for Instagram, TikTok, YouTube, and Twitter/X with benchmarks.

Email signature generator

Create professional HTML email signatures with live preview. Multiple styles, free, no signup.

Color palette generator

Generate harmonious color palettes using color theory. Complementary, analogous, triadic, and more.

Aspect ratio calculator

Calculate image and video aspect ratios instantly. Find the right dimensions for any platform.

Contrast checker

Check color contrast ratios for WCAG AA and AAA compliance. Ensure your text is accessible.

Lorem Ipsum generator

Generate placeholder text in paragraphs, sentences, words, or list items. Instant, free.

Markdown to HTML converter

Convert Markdown to clean HTML with a live side-by-side editor. Preview or copy the source.

Word counter

Count words, characters, sentences, and paragraphs. Get readability scores and keyword density.

Instagram Engagement Calculator

Calculate your Instagram engagement rate with benchmarks

TikTok Engagement Calculator

Calculate your TikTok engagement rate with benchmarks

YouTube Engagement Calculator

Calculate your YouTube engagement rate with benchmarks

Twitter/X Engagement Calculator

Calculate your Twitter/X engagement rate with benchmarks

Ready to stand out?

Create your page now. Ready in minutes.

Create your pageFree to try, no credit card required

Trusted by individuals and businesses worldwide