WordPress explained

Robots.txt

  • 1 min read
  • Term 95 of 143
A B C D E F G H I J K L M N O P Q R S T U V W X Y Z

A robots.txt file is a plain text file placed at the root of your website that tells search engine crawlers which pages or sections they’re allowed to visit and which to skip. It’s not a security tool, but a set of polite instructions that well-behaved bots follow.

What Robots.txt Looks Like

A basic example uses “User-agent: *” to target all bots, followed by “Disallow: /wp-admin/” to block the admin area. You can also specify specific bots by name, like Googlebot.

Common WordPress Entries

  • Block /wp-admin/ to keep the admin area out of search results
  • Allow /wp-admin/admin-ajax.php so front-end AJAX features still work
  • Block /wp-login.php to reduce spam bot traffic

Where to Find It

Visit yourdomain.com/robots.txt to see your current file. Yoast SEO and Rank Math let you edit your robots.txt from within the WordPress dashboard without using an FTP client.

Important Warnings

  • A mistake in robots.txt can accidentally block Google from crawling your entire site
  • Disallowing a URL doesn’t remove it from the index if other sites link to it
  • Use the noindex meta tag for pages you want controlled in results

Related: XML Sitemap

Related: Technical SEO

Ferdy.com

Learn more about Robots.txt

Want to see this in practice? These free tutorials show it step by step:

Want AI to do the work? Use the free SEO and AI search optimization prompt.

Still stuck?
Watch along

Most questions are easier when you see them on screen. Find the tutorial for your problem and follow along.

Black Friday A full year of Hostinger web hosting + free domain for about $30 with coupon code FERDY1 year hosting + free domain: about $30 with code FERDY Ends Dec 14 Get the deal Ad