Cloudflare’s compliant crawler highlights tension – and opportunity – in the emerging AI content market

March 19, 2026 Cloudflare is pushing to create a market for licensed AI content, offering a compliant crawler aimed at giving publishers more control and reducing inefficient site crawls — even as some question the irony of the company that once blocked scrapers now building one itself. Last week, the company released a crawl API (its crawl endpoint within Cloudflare’s browser rendering API) that can scrape an entire website with one request. Advertisement TOP STORY AI Revenue Generation Cloudflare's compliant crawler highlights tension – and opportunity – in the emerging AI content market  Cloudflare faces tension in its new role: sitting in the middle between publishers and AI companies while balancing trust, control and monetization.  OTHER THINGS TO KNOW When marketers implement mobile-grade measurement standards, they give AI a grounded identity source and behaviors from which to work. Without that, they may get faster, but not better, optimization. Partner insights from AppsFlyer. By adopting a more optimized flighting approach to ad spend, marketers can deliver more marginal ROI on their investments. This strategy ensures that they're not wasting any of their spending. Partner insights from Keen Decision Systems. At a moment when social media is failing its users and attention is the scarcest resource of all, audience participation is the opportunity publishers can no longer afford to ignore. Partner insights from Viafoura. MARKETING The Creator Economy Brands turn niche news creators into a new earned media engine Brands are shifting earned media strategies to tap niche news creators on social platforms. Brands in Culture The real winners of March Madness? Brands that move fast on NIL deals Companies across sectors, from footwear to personal care, are racing to sign college basketball players. PARTNER INSIGHTS FROM SEEDTAG How neuro-contextual intelligence shifts media planning strategies Audience strategy has historically focused on reaching predefined consumer segments based on assumed traits. Neuro-contextual intelligence helps media planners identify high-intent moments in real time — distinguishing between passive browsing and active consideration. MEDIA & PLATFORMS AI Revenue Generation Media Briefing: What to expect at the Digiday Publishing Summit, March 2026 edition  Execs from The Atlantic, Arena Group, Bloomberg, Business Insider, The Guardian, New York Post, People Inc., Washington Post, and more, will share their strategies on everything from zero-click audience strategy, to AI licensing deals and RAG readiness, to how they're embracing creator strategies to help boost engagement with younger audiences. PARTNER INSIGHTS FROM VIASAT ADS In-flight sets up retargeting for continuous campaigns with high engagement Sponsored Wifi is an effective way to reach consumers before a large event and enable retargeting. By presenting an ad and redirecting users to an advertiser's site, retargeting can occur post-event to keep up momentum. Advertisement PARTNER INSIGHTS FROM THE WEATHER COMPANY Video: How weather universally drives consumer behavior Weather impacts what people try, buy and how they feel. This powers targeting to predict the moments when weather shifts a consumer's mindset toward purchase, driving sales and long-term growth. Learn more in this video. ICYMI Member exclusive Future of TV Briefing: How Netflix is shaking up the upfront deal model This week's Future of TV Briefing looks at Netflix's single- and multi-year deals with advertisers that extend beyond the scope of traditional upfront commitments. Evolving Agencies After WPP reckoning: The case for and against principal media Ten years later, principal media is back in the zeitgeist. Why agencies and brands are clashing over the so-called hidden model. Written by Jim Cooper and Sara Jerde Were you forwarded this email? Sign up to get it daily. Advertise | Digiday+ | Events | Awards Update your email | Unsubscribe | Privacy policy