Definition
Robots.txt is Robots.txt is a plain text file at the root of a website that tells web crawlers which pages they may or may not visit, and it works only if the crawler chooses to follow it.
Robots.txt is a small text file that a website places at the root of its address, for example example.com/robots.txt. It lists rules for automated visitors, called web crawlers: which parts of the site they may visit and which they should stay out of. The convention is known as the Robots Exclusion Protocol, and it was formalized as an internet standard, RFC 9309, in 2022 after decades of informal use.
Think of a sign on a door that says "staff only." It tells honest visitors where not to go. It does not lock the door.
The file is made of short rules. Each group names a crawler by its user agent, such as a search engine's bot or an AI company's bot, and then lists paths it is allowed or disallowed to fetch. A rule can cover the whole site, one folder, or a single page. A site can give different instructions to different crawlers, and it can point to its sitemap.
A well-behaved crawler fetches robots.txt first, reads the rules that apply to it, and follows them. Nothing enforces this. A crawler that ignores the file can still request pages, so anything truly private needs a login or other access control, not a robots.txt line. The file is also public, so listing a secret folder in it advertises the folder.
Companies that build AI models gather text from the web, as do the products that search it for answers. That made robots.txt the main tool publishers use to say whether their pages may be collected for training data or fetched by an AI search feature. Many AI companies publish the name of their crawlers so site owners can allow or block them one by one.
Blocking has limits. A rule keeps a compliant crawler from fetching pages from now on, but it does not remove content already collected. It does not stop a user's browser agent acting on their behalf, since many treat that as a person browsing. Blocking a search crawler may also remove you from that product's results.
Open your robots.txt and read it as a reader would. Confirm that it names the crawlers you care about, that a broad rule is not blocking your search engine traffic by accident, and that nothing sensitive depends on it for protection. Review it again when an AI company or platform announces a new crawler or changes its terms, because the file is how your decision gets communicated.
Only if their crawler follows it. It asks compliant crawlers to stay away from listed pages going forward, but it cannot enforce that and does not remove content already collected.
No. The file is public and relies on crawlers choosing to obey it. Private content needs real access controls such as a login.
Meta removed a clause from its updated commercial terms that would have let it use content from sites tied to business accounts to train its products, TechCrunch reports. The crawling permission stays, and it takes effect Oct. 30.