How to Edit robots.txt in WordPress (With & Without Plugin)

Not having a strong web presence is like not existing at all. Your website is your storefront, your portfolio, and your voice.

Controlling how search engine crawlers discover and interact with it is crucial for its success.

In this guide, we will discuss the robots.txt file, a critical tool for managing your WordPress robots.txt file and optimizing your site for SEO. We’ll cover everything from the basics of robots.txt rules and directives (like Disallow and Allow) to editing your file using popular SEO plugins or even directly via FTP or cPanel.

But that’s not all! We’re going beyond traditional search engines.

In the age of AI, we’ll show you how to manage your website’s presence in tools like ChatGPT and Claude using specific robots.txt user-agent configurations.

You will also learn to control whether your content is used to train these powerful AI models, giving you unprecedented control over your digital footprint. This feature has become increasingly important with the advancement of AI, so understanding your robots.txt file and web robots in general is more powerful than ever.

By the end of this guide, you’ll be a robots.txt expert! 

What is robots.txt?

The robots.txt file is a plain text file in your website’s root directory (e.g., yourwebsite.com/robots.txt). It is a set of instructions for search engine crawlers (also known as bots or spiders) that tells them which parts of your site they shouldn’t access. It’s a tool for controlling how search engines interact with your website, but it’s not a security mechanism. It is a public file that any human or bot can see.

📖 Suggested read: 15 Best Performance Testing Tools to Improve Your Site

Purpose of robots.txt

The primary purpose of robots.txt is to manage crawler traffic, not to hide pages from search results entirely. While it strongly suggests which areas to avoid, it doesn’t guarantee exclusion from search indexes. Here’s a breakdown of its key functions:

• Preventing Crawling of Duplicate Content: Websites often have multiple URLs that lead to the same content (e.g., with and without “www” or with different URL parameters). robots.txt can help you indicate which version you prefer search engines to crawl and avoid duplicate content penalties.
• Blocking Access to Sensitive Directories: You might have areas of your website that are not intended for public consumption, such as your admin panel (/wp-admin/ on WordPress), staging environments, or directories containing internal files. robots.txt can tell crawlers to steer clear of these. However, remember that this is a suggestion, not an enforced restriction; for true security, use proper authentication and authorization methods.
• Conserving Crawl Budget: Large websites, especially e-commerce sites, can have thousands or even millions of pages. Search engines have a limited “crawl budget” – the amount of time and resources they’ll dedicate to crawling a particular site. You can ensure that crawlers prioritize indexing your most valuable content by strategically disallowing less important or dynamically generated pages (like certain search result filters).
• Specifying the Location of Sitemaps: robots.txt can include a directive pointing search engines to your XML sitemap(s). Sitemaps are lists of all the important URLs on your site, helping search engines discover and index your content more efficiently.

📖 Suggested read: 10 Best WordPress Management Tools To Easily Manage Multiple Websites

How Search Engines Use robots.txt

When a search engine crawler (like Googlebot) visits a website, it typically checks for the robots.txt file first. It parses the file and looks for instructions specifically targeted at it (via the User-agent directive).

1. User-agent: This line specifies which crawler the following rules apply to. User-agent: * means the rules apply to all crawlers. You can also target specific crawlers, like User-agent: Googlebot.
2. Comments: Lines starting with # are considered comments and will be ignored by web crawlers.
3. Disallow: This is the core directive. It tells the crawler not to access the specified URL path. For example, Disallow: /private/ would instruct crawlers to avoid the /private/ directory and everything within it.
4. Allow: This directive is used to override a Disallow rule. It’s helpful if you want to block an entire directory but allow access to a specific file or subfolder within it. For instance:

User-agent: *
Disallow: /images/
Allow: /images/logo.png

The above code snippet blocks access to the /images/ directory, except for the logo.png file. You should note that robots.txt is based on a cooperative system. Reputable search engines (like Google, Bing, etc.) generally respect the instructions in robots.txt. However, malicious bots or scrapers might ignore them completely.

Therefore, robots.txt should not be relied upon for security; use proper authentication and .htaccess rules for sensitive areas. Also, a disallowed page may still appear in search results if it is linked from other pages.

📖 Suggested read: How To Install WordPress With RunCloud | Step-By-Step Guide

Importance of robots.txt For AI Crawlers

While robots.txt has always been important for managing traditional search engine crawlers, its importance has grown significantly with the rise of AI-powered crawlers. These AI crawlers, used by companies like OpenAI, are designed to gather vast amounts of data from the web, often to train large language models (LLMs) and other AI systems. This introduces new considerations for website owners regarding data usage, privacy, and control.

Why is robots.txt More Important Now?

• Data Usage Control: AI models require massive datasets for training. Your website’s content, if crawled, could potentially be used to train these models. robots.txt allows you to express your preference regarding this usage. You can choose to allow AI crawlers to access your content, disallow them entirely, or selectively control access to specific parts of your site.
• Performance and Bandwidth: AI crawlers can be very aggressive and potentially overload your server. robots.txt can help you manage this by limiting their access or specifying crawl delays (though not all crawlers respect crawl-delay directives).
• Ethical Considerations: There are ongoing discussions and debates about the ethics of using publicly available data to train commercial AI models. robots.txt allows website owners to participate in this discussion by making their preferences known.
• Search Indexing: Some AI crawlers enhance search experiences within specific AI products. Controlling access to these can directly impact whether your content is discoverable through these AI-powered search interfaces.

How to Edit the robots.txt in WordPress

Two primary ways to edit your robots.txt file in WordPress are using a plugin (generally recommended for ease of use and safety) or editing the file directly (which requires more technical knowledge).

Method 1: Editing robots.txt with a Plugin

This is the recommended method for most users, especially beginners. Many SEO plugins offer built-in robots.txt editors, which provide a user-friendly interface and reduce the risk of syntax errors.

1. Install and Activate an SEO Plugin: You can use popular plugins such as Yoast SEO, Rank Math, All in One SEO Pack, SEOPress, etc. These plugins offer a wide range of SEO features, including robots.txt management.
2. Locate the robots.txt Editor: The exact location of the editor varies depending on the plugin. Refer to the official documentation of your WordPress plugin to follow along:
   • The robots.txt file in Yoast SEO
   • How to Edit Your Robots.txt With Rank Math SEO
   • Edit your robots.txt file – SEOPress
   • Using the Robots.txt Tool in All-in-One SEO
3. Edit the File: The plugin will typically provide a text area where you can view and edit the contents of your robots.txt file. Make your desired changes and exclude any unnecessary paths from crawling.
4. Save Changes: Click the “Save Changes” or similar button to apply your modifications.
5. Test your Changes: Use the Google search console’s robots.txt tester to ensure your edits work correctly.

Advantages of using a plugin:

• User-Friendly Interface: Simplifies and streamlines the editing process.
• Syntax Validation: Many plugins will check for basic syntax errors, reducing the risk of accidentally blocking important parts of your site.
• Integrated with SEO Tools: Often provides seamless integration with other SEO features within the plugin.
• Reversion Options: Some plugins offer version history or backup options, allowing you to revert to previous versions if needed.

Method 2: Editing robots.txt without a Plugin

If you don’t want to use an additional plugin, you can directly access and modify the robots.txt file on your server. However, this requires more technical expertise and, if not done carefully, carries a higher risk of errors.

1. Access Your Server: You’ll need to connect to your server using one of the following methods:
   • RunCloud File Manager: RunCloud provides a built-in file manager for creating and modifying text files on your server.
   • FTP (File Transfer Protocol): Connect to your server using an FTP client such as FileZilla. Your web hosting provider usually provides your FTP credentials (hostname, username, and password).
   • cPanel File Manager: Most web hosting providers offer cPanel, which includes a web-based File Manager. Log in to your cPanel account and navigate to the File Manager.
   • SSH (Secure Shell): Advanced users can use SSH to obtain command-line access to the server. This requires SSH credentials and familiarity with command-line tools.

2. Locate the robots.txt File: Navigate to the root directory of your WordPress installation. This is typically the directory where your wp-config.php file is located. If it doesn’t exist, you can create a new plain text file named robots.txt.
3. Edit the File:
   • RunCloud File Manager: Locate the file and click on it; this will open a text editor in a new browser window.
   • FTP: Download the robots.txt file to your computer and edit it using a plain text editor such as Notepad++, Sublime Text, or VS Code. You can’t use a word processor like Microsoft Word. After editing the file, save it and upload the modified file back to the server, overwriting the existing file.
   • cPanel File Manager: Right-click on the robots.txt file and select “Edit” or “Code Edit” (depending on your cPanel version). Make your changes directly in the web-based editor.
   • SSH: Use the nano or vim command-line text editors to edit your robot.txt file.

4. Save Changes: Ensure you save the changes you’ve made to the file. If using FTP, ensure the updated file is uploaded and overwrites the old one.
5. Test Your Changes: After editing your file directly, it is essential to test it. Use Google Search Console’s robots.txt Tester tool to verify that your changes are working as intended and that you haven’t accidentally blocked important content.

While editing the robots.txt file directly without a plugin can be more technically involved, it offers some distinct advantages for advanced users and those who prefer greater control over their website’s settings. 

📖 Suggested read: How Google Interprets the robots.txt Specification

This approach ensures that all the necessary features and tools are built directly into the core system. You have complete autonomy over the file’s content and structure without depending on external plugins or add-ons. This eliminates compatibility issues, potential security risks, and the need for additional software installations. 

However, it does have a few disadvantages:

• Higher Risk of Errors: Requires careful attention to syntax; mistakes can have significant negative consequences.
• Less User-Friendly: Requires technical knowledge of FTP, cPanel, or SSH.
• No versioning: If you make a mistake, it will be difficult to revert.

Unless you are comfortable with server-side file management, a plugin is strongly recommended for editing your robots.txt file in WordPress. If you do choose to edit directly, always test your changes thoroughly using Google Search Console’s robots.txt Tester.

📖 Suggested read: How to Fix WordPress Revisions Not Showing [SOLVED]

If you aren’t comfortable with command line tools but still want to edit your configuration files manually, then you should note that RunCloud users can manage and edit their robots.txt through a clean, intuitive interface, offering a safer alternative to direct FTP or SSH access while maintaining control.

📖 Suggested read: 3 Free Ways To Migrate WordPress From Shared Hosting To Cloud Server

Managing AI Crawlers with robots.txt

This section will focus on managing AI crawlers using robots.txt, specifically for OpenAI crawlers. By understanding these crawlers and their user agents, you can control their access to your website’s content.

OpenAI provides clear documentation on its crawlers, allowing you to make informed decisions about how you want them to interact with your site.

• GPTBot: This is OpenAI’s primary crawler for training its generative AI foundation models (like those powering GPT-3, GPT-4, etc.). Disallowing GPTBot signals that you do not want your website’s content used for training these models.

User-agent: GPTBot
Disallow: /

This example blocks GPTBot from crawling your entire site.

• OAI-SearchBot: This crawler specifically displays websites in search results within ChatGPT’s search features. Allowing OAI-SearchBot can help your site appear in these results.

User-agent: OAI-SearchBot
Allow: /

This allows OAI-SearchBot to crawl your site. You might combine this with a “disallow” command for GPTBot to appear in search results but prevent model training.

• ChatGPT-User: It is not used to crawl the web automatically. It is a proxy through which a user’s request is sent. It is not used for general training purposes.

User-agent: ChatGPT-User
Allow: /

You can combine rules for different crawlers in a single robots.txt file. For example:

User-agent: *
Disallow: /wp-admin/
Disallow: /private/
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /

The above example will:

1. Blocks all crawlers (including traditional search engines) from /wp-admin/ and /private/.
2. Blocks GPTBot from the entire site (preventing training data collection).
3. Allows OAI-SearchBot to crawl the entire site (enabling appearance in ChatGPT search).

While we provided an overview of OpenAI’s crawlers and user agents, we recommend reading AI / LLM User Agents: Blocking Guide to learn more.

📖 Suggested read: How to Easily Create a WordPress Staging Site in RunCloud

After Action Report: Mastering Your robots.txt File

Throughout this guide, we’ve shown you why robots.txt is a small but mighty file that manages how search engines and, increasingly, AI crawlers interact with your website. We discussed its core purposes: managing crawl traffic, conserving crawl budget, and specifying sitemap locations. We also explained that robots.txt is a suggestion, not a security mechanism.

Controlling your site’s interaction with crawlers is essential for a healthy web presence. You want search engines to index your valuable content efficiently, avoid duplicate content issues, and protect sensitive areas. With the advent of AI crawlers, you also need to consider how your content might be used for training AI models, and robots.txt gives you a voice in that process.

But managing files directly on your server, especially for tasks like editing robots.txt, can be intimidating. Traditional methods such as FTP clients and cPanel’s File Manager can feel clunky and risk errors if you’re not experienced.

This is where RunCloud shines.

RunCloud’s intuitive dashboard provides a built-in File Manager that simplifies server-side file management. No more juggling FTP clients or navigating complex cPanel interfaces!

• Direct Access: Access your website’s root directory and locate your robots.txt file.
• Built-in Text Editor: RunCloud’s File Manager includes a powerful yet easy-to-use text editor directly within the dashboard. You can make changes to your robots.txt file, save them, and then be done – all without leaving your browser.
• Secure and Streamlined: RunCloud’s interface provides a secure and streamlined way to manage server files, minimizing the risk of accidental errors.
• One-Click Staging: Create a staging environment to experiment with before pushing changes to the live server.

Ready for Effortless WordPress Management? Choose RunCloud.

RunCloud isn’t just about simplifying robots.txt management. It’s a complete WordPress hosting platform designed for speed, security, and ease of use. From deploying new sites with a single click to managing server configurations, backups, and security settings, RunCloud empowers you to take control of your WordPress hosting without headaches.

Here’s why RunCloud is the best choice for your WordPress site:

• Blazing Fast Performance: Optimized server configurations and caching mechanisms ensure your site loads incredibly quickly, improving user experience and SEO.
• Robust Security: Built-in security features, including firewalls and regular security updates, protect your site from threats.
• Effortless Management: An intuitive dashboard makes server management a breeze, even for non-technical users.
• Scalability: Easily scale your server resources as your website grows.
• Expert Support: RunCloud’s support team can assist you with any questions or issues.

Start your free trial today and see how easy WordPress hosting can be.

FAQs on robots.txt in WordPress