Definition
Web crawler is a web crawler is an automated program that visits web pages and follows links to read and collect their content, used by search engines and by AI companies gathering data to train or ground models.
A web crawler, also called a bot, spider or scraper, is software that loads web pages automatically. It starts with a list of addresses, reads each page, finds the links on it, and follows them to more pages. Repeat that across billions of pages and you have a copy of a large part of the web.
Think of a very fast librarian who walks every aisle and writes down what is on each shelf.
Crawlers have long powered search engines. They now serve AI too, in a few different ways:
Which of these a given crawler does is not always obvious from its name, which is why site owners often ask companies to say plainly what each crawler is for.
Every crawler should identify itself with a name, called its user agent. Site owners publish a file called robots.txt at the root of their site listing which crawlers may visit which paths. The format is standardized as the Robots Exclusion Protocol (RFC 9309).
The important caveat comes from the standard itself: robots.txt is not access authorization. It is a set of requests that well-behaved crawlers choose to honor. It does not technically stop anyone. Content that must stay private needs real access controls, such as a login.
Other tools exist too, including blocking by network address, rate limiting, and terms of service, but none is perfect against a determined bot.
If you publish anything online, crawlers decide how your work is found, indexed and possibly learned from. Blocking all of them can remove you from search. Allowing all of them can feed your content into training data for models you never agreed to help. Most owners choose per crawler.
If you build tools that fetch pages, you are running a crawler. Identify it clearly, respect robots.txt, and slow down so you do not overload other people's servers.
Content a crawler reads can also carry hidden instructions aimed at AI systems, which is prompt injection. A browser agent is different from a crawler: it acts for one user in a real browser rather than sweeping the web in bulk.
No. It is a request that well-behaved crawlers choose to follow, not a lock. Private content needs real access controls such as a login.
A crawler discovers pages by following links. A scraper extracts specific data from pages. The words overlap, and one program often does both.
Meta removed a clause from its updated commercial terms that would have let it use content from sites tied to business accounts to train its products, TechCrunch reports. The crawling permission stays, and it takes effect Oct. 30.