What Is a Robots.txt Generator?

A Robots.txt Generator helps you create the plain-text rules that tell compliant web crawlers which parts of a website they may or may not crawl. Instead of writing every directive manually, you can choose a user agent, add Allow or Disallow paths, include sitemap URLs, and copy or download a ready-to-review robots.txt file.

This online robots.txt generator works locally in your browser. It does not crawl your website or publish changes for you, so you stay in control of the final file before uploading it to your site.

How to Create a Robots.txt File

1. Choose a quick-start option or keep Allow all crawlers as the starting point.
2. Use * for all crawlers or select a specific user agent such as Googlebot.
3. Add any paths that should be allowed or disallowed for that crawler group.
4. Add one or more complete sitemap URLs if you want to reference your XML sitemap.
5. Review the generated file and the Rule Check messages before publishing.
6. Copy the text or download it as robots.txt, then place the file at the root of the host it applies to.

For example, a file for https://example.com/ normally belongs at https://example.com/robots.txt. A robots.txt file placed inside a subfolder does not control the whole site.

Robots.txt Directives Explained

User-agent

The User-agent field identifies which crawler a group of rules applies to. An asterisk (*) means the group is intended for all crawlers that do not have a more specific matching group.

User-agent: *

Disallow

Disallow tells the selected crawler not to crawl a matching URL path. Paths are case-sensitive, so /Private/ and /private/ can be different locations.

User-agent: *
Disallow: /private/

Allow

Allow can create a crawlable exception inside a broader blocked path. When multiple rules match, Google uses the most specific matching rule; if conflicting rules are equally specific, the less restrictive rule is used.

User-agent: *
Disallow: /admin/
Allow: /admin/public-page/

Sitemap

The Sitemap directive points crawlers to a complete sitemap or sitemap-index URL. The value should be an absolute URL including the protocol and host.

Sitemap: https://example.com/sitemap.xml

Robots.txt Examples

Allow Crawling Across the Site

If a crawler group has no blocking rules, there are no crawl restrictions from that group.

User-agent: *

Block One Directory

User-agent: *
Disallow: /private/

Block the Entire Site

User-agent: *
Disallow: /

This is a powerful rule and should only be used intentionally. A site-wide block can prevent compliant search crawlers from fetching the website.

WordPress Starting Example

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

This is a common editable starting pattern, not a requirement for every WordPress website. Your actual file should reflect the paths and crawling needs of your own site.

Wildcards and URL Matching

Robots.txt paths can use a limited set of matching characters. The asterisk * can match zero or more characters, while $ can mark the end of a URL. For example, the following rule can match URLs that end in .pdf:

User-agent: *
Disallow: /*.pdf$

Specificity matters. If more than one Allow or Disallow rule matches a URL, the longer matching rule normally takes precedence for Google. That is why the generator includes a local path preview rather than relying on a simple first-match calculation.

Robots.txt Controls Crawling, Not Indexing

A robots.txt file generator is useful for managing crawler access, but robots.txt is not a universal indexing control. If a blocked URL is discovered through links or other sources, Google may still know about the URL and can potentially show the URL in search results without crawling its page content.

For that reason, do not use robots.txt as a replacement for an indexing directive when your actual goal is to keep a page out of search results. Also avoid placing a noindex directive inside robots.txt and expecting Google to use it as a supported robots.txt rule.

Robots.txt Is Not a Security Tool

Do not use Disallow rules to protect passwords, customer data, staging credentials, private documents, or other sensitive information. Robots.txt is publicly accessible and compliant crawlers may follow it, but it does not stop a person or non-compliant bot from requesting a URL directly.

Private resources should be protected with proper authentication, authorization, server configuration, or other access controls.

Where Should the Robots.txt File Be Uploaded?

The file should be named exactly robots.txt and placed at the top level of the host it controls. The rules are scoped to that host, protocol, and port. For example, a file at https://www.example.com/robots.txt applies to that HTTPS www host; it does not automatically control a different subdomain or protocol.

Google also expects robots.txt to be served as plain text and supports UTF-8. Google currently processes up to 500 KiB of a robots.txt file, so extremely large files should be consolidated where practical.

Common Robots.txt Mistakes to Avoid

• Accidentally blocking the entire site: Disallow: / is a site-wide crawl block for the matching user agent.
• Using robots.txt for private data: crawler instructions are not access control.
• Assuming Disallow guarantees deindexing: crawling and indexing are related but different processes.
• Using unsupported Google directives: do not rely on crawl-delay, noindex, or Host as Google robots.txt controls.
• Using a relative sitemap value: use a complete URL such as https://example.com/sitemap.xml.
• Ignoring case: Allow and Disallow path values are case-sensitive.
• Blocking resources needed for rendering: review CSS, JavaScript, image, and other resource paths before restricting them.
• Publishing without testing important paths: check representative URLs and crawler groups before deploying changes.

Why Use an Online Robots.txt Generator?

A free robots.txt generator is useful when you want valid, readable crawler rules without repeatedly typing the syntax by hand. It can also make it easier to review multiple user-agent groups, spot duplicate rules, create sitemap directives, and download the final plain-text file.

The AddLinker robots.txt maker also provides a local rule preview for the file you are building. It does not fetch a live robots.txt file or verify server configuration, so always review the deployed file on your own website after publishing.

Frequently Asked Questions