Cloudflare just segmented AI traffic into three types: Search, Agents, and Training, and they’re letting their customers turn access on or off for each type. That’s better than a single on/off switch for bots, but for a high-end website that isn’t enough control.
The real control businesses need is for each service within those categories. They need to decide which crawler gets which content, for what purpose, and under what terms.
This isn’t hypothetical. One of WordPress VIP’s enterprise customers might want to allow every bot to access their site except Perplexity, because they’re negotiating a deal with them. Another maybe has a deal with Perplexity and loves them, but needs to block OpenAI. And many customers just want all bots redirected to TollBit to collect micropayments for accessing their content.
Giving three on/off switches isn’t enough. Enterprise customers need fine-grained controls, down to the individual service.
I did 3 hours of back-to-back Zoom calls from my local Starbucks this morning. There were a few moments where I felt like I was making people suffer through some background noise to indulge me and my caffeine addiction, but it could have been crazier, right?
For decades, ranking on Google was survival. Now ADWEEK reports that some of the biggest media businesses are not-so-quietly making plans to disappear from it.
This isn’t just one angry newspaper. The Wall Street Journal lists USA Today, Reuters, Politico, The Economist and People Inc. among the publishers who are considering cutting off Google. And Cloudflare, which sits in front of a fifth of the web, is flipping its defaults to block the crawlers that scrape for both search and AI training.
What changed?
Publishers have always known this dependence was a risk. This is not a revelation. What changed is that the relationship has stopped being a two-way exchange. It’s lopsided.
Neil Vogel, CEO of People Inc., calls Google an “intentional bad actor” for using the same crawler for both search and AI, but he admits People still can’t block it without losing the search traffic it depends on. People Inc is “probably heading towards more confrontation than productivity.”
(I’ve watched Vogel speak a few times recently and he inspires confidence. He seems to have a firm grasp of the industry and an actual plan.)
This Google trap is why the open web matters. It isn’t because of nostalgia or a blind love of all things open source. It’s because ownership and portability increases resilience.
Traffic comes and goes. Ownership compounds.
But the sharp media newsletter A Media Operator makes the case that “No One is Going to Block Google” and the math backs them up. When a publisher still gets over 40% of its traffic from search, walking away from Google is a top-line bloodbath. Google’s Liz Reid reinforces the symbiotic relationship: “Search is not a useful product if the web is not a thriving product.” And Future’s Mike Peralta asserts: “[AI is] nothing but a hallucination without our content.” The industry is also too fragmented for everyone to leave Google all at once, and that’s the only version where blocking actually moves Google.
So will publishers push the big red button? I doubt it, and it doesn’t really matter if a few of them do.
The preparation is the actual prize here. Building newsletters, direct audience, licensing revenue, and owned infrastructure makes you stronger whether you ever block a single crawler or not. Silicon Valley has a word for this: antifragile, systems that get stronger under stress.The exercise of getting ready to survive without Google is exactly what makes you the kind of business that never has to.
The goal isn’t to leave Google.
The goal is to build a business that could if it has to, and no longer has to care whether it does.
AI isn’t replacing websites. But it IS changing what they’re used for.
For years, websites have been the destination.
Today, they’re your system of record that powers AI interactions everywhere else.
I got to join Lauren Goerz on The Dialogue Architects Podcast to explore all the ways that AI is reshaping enterprise content management, digital experiences and the future of work.
We covered enterprise websites as the source of truth in an AI-first world, how answer engines are changing content discovery and totally disrupting traditional customer buying journeys, and why governance and accessibility matter even more as adoption grows. We also talked about jobs to be done (JTBD) as a better way to divide work between people and agents, and what conversational interfaces could mean for enterprise software.
Thanks Lauren for a fun, wide-ranging conversation. You do great research and run a fantastic show.
Our big “Future of the Web” research report says that people hate sites that are robotic. They hate digital experiences, things that feel synthetic.
Does it really need to be like this?
The first sites I built in the ‘90s were all by hand. There were no CMSes. Then we hooked up databases to websites. In 2000, browsers added a way to let people make content bold and add links and images. From there, filling websites became a big machine, an assembly line. That’s who I work with. Large sites with large teams making thousands of new stories every day for millions of visitors.
You NEED a machine for that.
Heck, if you want to compete, you NEED AI for that.
But does it have to feel like robots made it?
Two-thirds of people can’t name a single brand that’s doing AI well in their own messaging. They can’t name one.
So the bad news is: everything is terrible.
I guess the good news is: maybe YOU can be the one who isn’t terrible?
And if you do manage to cut through all of the AI slop and reach people with a story that’s organic and human, one that feels like a person made it, maybe your audience will stick around and your business will win.
Your developers are completely AI-pilled. They live in Claude Code, Cursor, Windsurf. But your security model doesn’t.
That’s why Secure MCP matters for WordPress VIP. Your agents run as a signed-in VIP Dashboard user. They can only do what you allow them to do. Trying to perform sensitive operations or accessing high-risk resources only happens if you pass our Step-up Authentication. Every command you run gets recorded in our immutable MCP Call Log. Secure MCP gives teams more than 50 platform tools through one governed endpoint: apps, environments, domains, logs, metrics, users, and WordPress content workflows too.
And there’s a kill switch that pauses all access in under 60 seconds.
That kill switch is really important, because AI governance is meaningless if it can be ignored.
Are you old enough* to remember Google’s “I’m Feeling Lucky” button?
Originally, you had two choices with Google search. You could get a page with 10 links or if you were feeling lucky, you would just be taken to the top result. But there is no single top search result for a query anymore. Google serves you different results depending on where you are, who you are, what browser you’re using, what time of day it is.
There is no one answer.
AI answers are following that same pattern. According to a massive research study WordPress VIP just did about AI usage, most people still click through to the source after an AI answer, because the trust isn’t there yet.
But those clicks are just a phase.
The trust gap is closing faster than anything before it. It took Google more than twenty years to earn “I’m Feeling Lucky,” to get to the point where we just trust it to answer our questions for us and no one clicks any links. Publishers call this Google Zero, where Google no longer sends them any traffic.
When your search and AI traffic goes to zero, will your business survive? Do you have a direct connection to your audience? To your customers?
* The button still exists if you visit google.com on a laptop, but you’ll never see it when you search on a mobile browser or when you search from a URL bar. In fact, the button now cycles between feeling “lucky” and “hungry” and “playful” and “adventurous.” It’s a relic.
No one is making AI make sense to regular people. No one can tell them why it matters, why it’s good for them.
So what can a marketer do when there’s no incumbent, no playbooks, no guarantees?
Well, that’s the opportunity. No template to copy means no one is ahead of you. The first brand that says something true about AI plainly, in language their customers will actually repeat, wins the hearts and minds of the next big market.
If someone asked you what brand is doing AI messaging well, who would you name?
Every team I talk to has the same instinct for their content: go where the audience is. Chase them on search. Chase them on social. Now they’re chasing them on AI answer engines.
Whether you think that’s lazy or not, it can be valuable in the short term. But long term, it’s a trap.
Your only goal is to get a strong, direct relationship with your audience and your customers.
It’s the same with the software you choose for your site. If you build on a closed system, you’re stuck. You’re dependent.
One of my favorite publishers has consolidated two dozen sites onto WordPress VIP in our global private cloud. Every new site they acquire gets migrated in. Because they know that they are in control of their future. They own the stack. Everything they do is portable. If we don’t give them what they need, they can move to other WordPress hosts or even self host.
My indie web friends have been saying this for decades. But it’s not about nostalgia or freedom or being part of a movement. It’s about survival.
At this time of year? At this time of day? In this part of the country? Localized entirely within your WordPress?!?
You bet. It’s here now.
People joke that the “S” in MCP is for security. So WordPress VIP made Secure MCP for the biggest and most important websites on the planet. Auditable. Governed. Available inside all the apps you’re using to manage your sites today — Claude Code, Cursor, ChatGPT, Codex — and even apps that don’t start with a C.
My favorite feature is Step-up Authentication, where sensitive operations require you to authenticate again.
Brian Alvey 