Meta delays crawl terms for business sites to Nov. 20

Meta blinked on the date. It did not drop the crawler.

Meta has moved the effective date of its updated Commercial Terms and Self-Serve Ad Terms from Oct. 30 to Nov. 20, Engadget reported on Oct. 24. A Meta spokesperson told Reuters and Fast Company the company is "extending the effective date" to give itself more time to explain the changes to partners, according to Engadget.

What the terms still say

The clause at issue lets Meta crawl the websites tied to a business account. Meta's preview of the new terms, now dated Nov. 20, says: "You authorize us (or any entity acting on our behalf) to use automated software" to "access, index, cache, analyze, collect, retrieve, or crawl the domains, web pages, or URLs" linked to your products or services. It also lets Meta pull updates from those pages periodically.

The first version went further. It said Meta could use that content to "provide, develop, improve, research, train, and maintain the integrity of" its products, Engadget reported on Oct. 19. After the backlash, Meta removed "train" and limited the purpose to advertising and business products.

Meta spokesperson Andy Stone said the company "clarified" the language so the terms apply "only to our ads and business products," and that the changes "were not about scraping news sites," per Engadget.

Publishers did not buy the edit

The News/Media Alliance sent Meta CEO Mark Zuckerberg an open letter on Oct. 22 demanding Meta withdraw the terms and remove the crawling provision. "Publishers cannot practically disallow or block Meta's crawlers," it argues, because blocking would cost them visibility as business account holders. The letter calls the move "an exercise of market power" and anticompetitive.

Engadget reports that affected businesses have also filed complaints with antitrust authorities.

Meta says it will explain how its web crawlers work and make clear that site owners can use robots.txt to signal their preferences.

Robots.txt is a request, not a lock. A crawler follows it only if it chooses to.

What to do before Nov. 20

If you run a business page on Facebook or Instagram, these terms cover the website you link to it. That is most small businesses, and every publisher.

Read the clause yourself. It is short.

Check your robots.txt and see what it allows Meta's crawlers to do today. If you want them out of part of your site, say so there, and know it is a signal, not an enforcement.

If you publish content you license or sell, talk to whoever handles your contracts before the date arrives. The question is whether keeping a business page means granting this access, and Meta has not said otherwise.

Watch what Meta explains

Meta now has four weeks to explain its crawlers to partners. Watch whether the clause changes again before Nov. 20, and whether any antitrust regulator responds to the complaints.

Read the terms. Then check your robots.txt.