New classifications, enhanced analytics, and industry-defining partnerships will help site owners and transparent AI companies work together to build a thriving Agentic Internet
Cloudflare, Inc. (NYSE: NET), the leading connectivity cloud company, announced today new classifications, enhanced analytics, and industry-defining commercial partnerships that bring together site owners and transparent AI companies so that the agentic Internet can flourish. Cloudflare's new tools and integrations help site owners and AI companies optimize for discoverability, efficiency, and monetization. By establishing these new rails, Cloudflare is helping provide the foundation for a healthy, collaborative agentic economy that benefits society, site owners, and AI companies.
The fundamental nature of Internet traffic has shifted. Today, automated agents and bots drive more than half of all web requests. AI is becoming the new interface for accessing information and conducting commerce online, changing the rules for how content is discovered. Most website owners want their content in AI systems. But for businesses built on advertising or subscriptions, they are threatened by AI systems training on their content without compensation. We believe that if you're a business that wants your content in AI systems, we should make it as easy and efficient as possible; but if you're a business where you do not, then you should have the tools to restrict AI's access. In response, Cloudflare is testing new default classifications, delivering deeper insights to customers, making AI search faster, and ensuring creators are compensated when their content powers an answer. The result is a level playing field. It optimizes crawling efficiency, protects intellectual property, and allows creators and AI to thrive in the agentic economy.
"Last year we provided site owners with transparency and control over what bots access their content, and we are thrilled with the benefits it has had to the ecosystem," said Matthew Prince, co-founder and CEO of Cloudflare. "Now that the majority of traffic on the Internet is non-human, we must go further and act faster so that a sustainable ecosystem can emerge. Cloudflare's new tools and partnerships give website owners increased visibility and commercial opportunities and benefit AI companies that have bots with clear and transparent intent. We hope that our proposed default changes encourage mixed-use crawlers to separate out search from agent use and training."
A Deadline to End the Free Pass for Mixed-Used Crawlers
After thousands of conversations with AI companies and site owners, one thing is clear: most site owners want to be discoverable in AI. However, for those who monetize via ads and/or subscriptions, maximizing discovery shouldn't mean giving away intellectual property for free. Transparent AI companies have enabled this by clearly separating and labeling their bots by intent. However, mixed crawler bots disadvantage both site owners and transparent AI companies by blending search, agent use, and training. This forces content owners to choose between being discoverable in an agentic era and giving away their most valuable content without compensation. Today, the largest search engine has access to about 2X more information than leading AI companies because they make it difficult for customers to remain discoverable without also being used for AI.
Over the next two months, Cloudflare will engage with the ecosystem, listen to feedback, and run tests to finalize new defaults and classifications with a deadline of September 15, 2026.
For new customers and for new sites for existing customers, on September 15, 2026, the defaults will be set to allow for search but block training and agent use for pages with ads. Mixed crawlers that do not give site owners the ability to choose between search, agent use, and training, will be blocked on all pages with ads. At any time these settings can be changed by customers in their dashboard.
On September 15, 2026 these changes will also be made for all existing free customers that have not changed their settings by September 15, 2026 in their dashboard. These changes aim to level the playing field between AI companies, who are redefining what it means to be discoverable in an agentic era in an equitable and transparent way, and legacy search companies.
Enhanced Analytics to Drive Visibility and Monetize Content
The market for monetized content is here, with publishers and AI platforms signing more than 50 major content licensing agreements over the past year alone. To accelerate these arrangements, Cloudflare is introducing the new Attribution Business Insights dashboard, stripping away the data asymmetry that has long left content owners negotiating in the dark. For the first time, business stakeholders, not just security and IT teams, can clearly see how AI bots consume their content and track how much human traffic specific AI companies actually send back.
But understanding how AI consumes content is only the first step. As discovery shifts from search engines to answer engines, customers need to perfect a new discipline: Answer Engine Optimization (AEO). AEO is about understanding not just who is crawling, but how your content is cited in AI-generated answers, how often, and how prominently, across models. Just as web analytics became essential infrastructure for the search era, Cloudflare aims to provide the definitive measurement and optimization layer for the AI era.
Smarter Search and Fair Compensation
Negotiated deals can work for the largest publishers, but they don't scale to the rest of the web. Cloudflare is working on two fronts that help creators and AI companies alike: making AI search more efficient, and paying creators automatically when their work is used. Today, AI models and search engines repeatedly crawl millions of unchanged web pages searching for fresh data to answer user queries. Cloudflare data suggests that over 50% of crawl traffic from AI crawlers is spent re-fetching unchanged pages. The result is wasted bandwidth for publishers, wasted compute for AI companies, and no better answers to show for it. Because Cloudflare can see what has actually changed, it can signal when a page is worth re-fetching and when it isn't cutting needless crawls in the process, while delivering better results. Cloudflare is testing these signals with leading AI companies to measure how much they improve freshness and reduce wasted crawling, and plans to make them broadly available later this year.
A year ago, on the first Content Independence Day, Cloudflare launched the first version of Pay Per Crawl so publishers could charge AI companies to crawl their content. Today, Cloudflare is evolving Pay Per Crawl into Pay Per Use, so publishers are paid when their content actually creates value, not just when it's fetched. The company is working with top partners, including Ceramic.ai and You.com to turn that principle into a working market. With Ceramic.ai, when a publisher opts in, they are paid every time their content appears in Ceramic's AI search results, and they get back the queries, citations, and ranking back. Other AI companies can bring the model that fits how they work You.com, for instance, will let an agent pay on demand for a specific piece of premium content it needs, while Cloudflare provides the common layer beneath them all.
A Year of Content Independence: What's Launched
Last year's Content Independence Day was fundamentally about defense-giving you the tools to take back control of your data. Over the past twelve months, Cloudflare has been aggressively building the infrastructure required to shift from mere protection to active enablement. Today's announcements represent the culmination of that foundation, delivering on three pillars for a sustainable ecosystem: control, transparency, and monetization. Last year, Cloudflare introduced AI Crawl Control, which set a permission-based model for AI crawling. To democratize these protections, Cloudflare partnered with platforms like beehiiv, providing independent creators and major media outlets alike with granular dashboards to easily track, block, or opt into specific AI models. To fundamentally secure agent interactions, Cloudflare also spearheaded Web Bot Auth, an open framework that allows AI bots and autonomous shopping agents to verify their identity and legitimacy to publishers and merchants. Cloudflare has also collaborated with leading payment providers to lay down the underlying payment rails for the emerging agentic commerce economy. Building directly upon Web Bot Auth, Cloudflare is helping them ensure AI shopping agents can transact safely at millions of merchants globally without exposing raw credentials. Ultimately, these initiatives help to provide the backbone for a sustainable agentic economy, ensuring the next era of the Internet is built on transparency, discoverability, and fair compensation.
Leading Publishers and AI Companies Support the Agentic Internet
- beehiiv: "We are entering a new era where content discoverability is being completely redefined and independent creators deserve to be at the forefront of that shift. The future is about giving creators control over their content and audience. We want to hand writers the steering wheel so they can easily opt into the new, agentic internet, maximize their visibility in next-generation answer engines, and actively scale their audiences on their own terms." Tyler Denk, CEO
- Ceramic.ai "To scale the future of AI search, we need a partner with massive reach and a shared commitment to transparency and fair compensation. Cloudflare allows us to easily and programmatically scale our operations. By bringing our pay-per-query model to their network, we ensure millions of content owners can seamlessly opt in to be compensated every single time their content appears in our search results." Anna Patterson, founder and CEO
- Condé Nast: "Over the past year, Cloudflare has continued to support premium publishers like Condé Nast, by being an advocate for ensuring we are paid for our content as critical inputs to the AI ecosystem. They provide market leading technical infrastructure and continue to be a thought partner for what's coming next." Geoff Campbell, SVP Strategy Business Development
- Patreon: As AI agents become increasingly powerful and popular, creators deserve a meaningful say in how their work is used by AI companies. On most of the Internet, creators have to accept AI training on their work just to reach and grow an audience. Patreon has a different vision: creators should be able to grow their audience and control how their work is used. That's why we're building on our existing work with Cloudflare to block known AI training crawlers at the network level across Patreon, while still allowing the crawlers that help creators get discovered and grow their businesses through search." Drew Rowny, SVP of Product
To learn more about Cloudflare's vision, please visit the following resources:
- Blog: Content Independence Day, one year on: building the business model for the agentic Internet
- Blog: Your site, your rules: new AI traffic options for all customers
- Blog: Unmasking the crawls with Attribution Business Insights
- Blog: Making AI Search smarter
About Cloudflare
Cloudflare, Inc. (NYSE: NET) is the leading connectivity cloud company. It empowers organizations to make their employees, applications and networks faster and more secure everywhere, while reducing complexity and cost. Cloudflare's connectivity cloud delivers the most full-featured, unified platform of cloud-native products and developer tools, so any organization can gain the control they need to work, develop, and accelerate their business.
Powered by one of the world's largest and most interconnected networks, Cloudflare blocks billions of threats online for its customers every day. It is trusted by millions of organizations from the largest brands to entrepreneurs and small businesses to nonprofits, humanitarian groups, and governments across the globe.
Learn more about Cloudflare's connectivity cloud at cloudflare.com/connectivity-cloud. Learn more about the latest Internet trends and insights at radar.cloudflare.com.
Follow us: Blog X LinkedIn Facebook Instagram
Forward-Looking Statements
This press release contains forward-looking statements within the meaning of Section 27A of the Securities Act of 1933, as amended, and Section 21E of the Securities Exchange Act of 1934, as amended, which statements involve substantial risks and uncertainties. In some cases, you can identify forward-looking statements because they contain words such as "may," "will," "should," "expects," "explores," "plans," "anticipates," "could," "intends," "targets," "projects," "contemplates," "believes," "estimates," "predicts," "potential," or "continue," or the negative of these words, or other similar terms or expressions that concern Cloudflare's expectations, strategy, plans, or intentions. However, not all forward-looking statements contain these identifying words. Forward-looking statements expressed or implied in this press release include, but are not limited to, statements regarding the capabilities and effectiveness of Bot Defaults, Attribution Business Insights dashboard, Pay Per Use and Cloudflare's other products and technology, the benefits to Cloudflare's customers from using Bot Defaults, Attribution Business Insights dashboard, Pay Per Use and Cloudflare's other products and technology and from integrating Ceramic.ai and You.com and Cloudflare's other products and technology with products of Ceramic.ai and You.com, the potential opportunity for Cloudflare to attract additional customers and to expand sales to existing customers through Cloudflare's product integrations with Ceramic.ai and You.com, the timing of when Bot Defaults, Attribution Business Insights dashboard, Pay Per Use or any of its related features will be generally available to all current and potential Cloudflare customers, the timing of whenBot Defaults, Attribution Business Insights dashboard, Pay Per Use or any of its related features will be developed and available in beta form, or generally available, to all current and potential Cloudflare customers, Cloudflare's technological development, future operations, growth, initiatives, or strategies, and comments made by Cloudflare's CEO and others. Actual results could differ materially from those stated or implied in forward-looking statements due to a number of factors, including but not limited to, risks detailed in Cloudflare's filings with the Securities and Exchange Commission (SEC), including Cloudflare's Quarterly Report on Form 10-Q filed on May 8, 2026, as well as other filings that Cloudflare may make from time to time with the SEC.
The forward-looking statements made in this press release relate only to events as of the date on which the statements are made. Cloudflare undertakes no obligation to update any forward-looking statements made in this press release to reflect events or circumstances after the date of this press release or to reflect new information or the occurrence of unanticipated events, except as required by law. Cloudflare may not actually achieve the plans, intentions, or expectations disclosed in Cloudflare's forward-looking statements, and you should not place undue reliance on Cloudflare's forward-looking statements.
2026 Cloudflare, Inc. All rights reserved. Cloudflare, the Cloudflare logo, and other Cloudflare marks are trademarks and/or registered trademarks of Cloudflare, Inc. in the U.S. and other jurisdictions. All other marks and names referenced herein may be trademarks of their respective owners.
View source version on businesswire.com: https://www.businesswire.com/news/home/20260701569514/en/
Contacts:
Cloudflare, Inc.
Daniella Vallurupalli
Vice President, Head of Global Communications
press@cloudflare.com
