Cloudflare updates robots.txt for the AI era – but publishers still want more bite against bots

September 29, 2025 Presented by As publishers continue their face off with AI bots scraping their content, they want solutions with real teeth. So while Cloudflare’s much-needed robots.txt update, which gives publishers more control over consent on how to use their content by AI crawlers, it doesn’t mean AI firms will play ball. And as AI transforms the way people search and shop online, marketers are racing to adapt, but fragmented tech stacks and unclear metrics are hurdles they’re having trouble clearing. TOP STORY AI Revenue Generation Cloudflare updates robots.txt for the AI era – but publishers still want more bite against bots Cloudflare's new Content Signals Policy effectively upgrades the decades-old honor system and adds a way for publishers to spell out how they do (and perhaps more importantly – how they don't – want AI crawlers to use their content once it's scraped.)  Advertisement OTHER THINGS TO KNOW Join this virtual event today at 1 p.m. ET to hear a guest speaker from Forrester unpack how brands and agencies are deciphering the right signals to skip the guessing game and drive results. Partner insights from Epsilon. At least 70% of teens do additional research before making a purchase, according to a TeenVoice survey. They weigh price, quality and social proof before making a purchase. Partner insights from TeenVoice. Brands that want to move faster, stretch budgets or measure impact across a wider set of objectives are using incrementality to uncover powerful and unexpected signals of campaign impact. Partner insights from Criteo. MARKETING Generative AI How AI is rewriting the rules of search and shopping – by the numbers As AI transforms the way people search and shop online, marketers are racing to adapt, but fragmented tech stacks and unclear metrics are slowing them down. The Creator Economy Creators brace for AI bots scraping their work  Six content creators and creator talent managers told Digiday that AI scraping of their content was an increasing concern going into the end of 2025, with three taking specific steps to block AI bot traffic and legally defend against the scraping of copyrighted content.  PARTNER INSIGHTS FROM MEDIA.NET How SSPs are evolving to better deliver transparency and performance Modern SSPs are going beyond display to help advertisers reach audiences wherever they are — whether it's CTV, digital OOH, mobile, desktop or audio. DIGIDAY EVENTS Connect with publishing leaders from The Telegraph, Reuters and more Limited spots remain to attend the Digiday Publishing Summit Europe, October 27-29 in Lisbon. Don’t miss out on this opportunity to network with top publishing leaders from Le Monde, the BBC and many others. Secure your pass before the meeting scheduler opens on October 23 to have your first choice of companies to meet with one-on-one.   Register MEDIA & PLATFORMS Member exclusive Media Buying Briefing: Gen AI creative emerges as new battleground between TV and platforms Broadcasters and walled gardens are competing to lower the cost of ad creative, as AI enables small business advertisers to make ads cheaply. Google on Trial What's in and what's out in the Google ad tech antitrust reckoning  Here's a definite (if not exhaustive) rundown of what's actually in play as the case moves into its next chapter. PARTNER INSIGHTS FROM CEROS Mood boards and coloring breaks: An inside look at a content strategy built on engagement B2B marketers can follow the lead of publishers like Her Campus and add micro-interactions like sliders or quizzes to case studies or solution pages to turn passive viewers into active participants. Advertisement DIGIDAY + Member exclusive Watch every session from the Digiday Publishing Summit, Fall 2025 Full video recordings of every session from the latest Digiday Publishing Summit are available exclusively for Digiday+ members. PARTNER INSIGHTS FROM META Advertisers are turning to Meta platforms to drive holiday sales According to a 2024 holiday shopping study by YouGov, commissioned by Meta, 58% of shoppers said Meta is likely to influence their purchases — up from 55% in 2023. Written by Jim Cooper and Sara Jerde Were you forwarded this email? Sign up to get it daily. Advertise | Digiday+ | Events | Awards Update your email | Unsubscribe | Privacy policy