This is an era of the “rug pull”, an age of the “pump and dump”.
The terms come from investment scams, but now apply more generally. A project is going well, prices are skyrocketing, attracting more investors: “the pump”.
Suddenly, it all comes crashing down. The rug is pulled and the founders vanish with the money. Investors are left with worthless assets: “the dump”.
After years of being courted by tech companies and encouraged to provide content optimised for their platforms and websites, news sites are increasingly finding themselves locked out and denied access to their users.
This is sometimes presented as a media industry issue, as a story about news sites outflanked by changing technology.
But it’s about much more than that. News sites are the canaries in the coalmine.
This is about the biggest change to the web in 20 years, one that will affect how you access information and what you read and watch.
Most of us have grown up with the “open web” — the enormous global collection of websites where information and content are published and shared freely without requiring the approval of a hypothetical web-owner or third party.
It’s so commonplace it’s hard to imagine a future without it, and yet it’s under threat.
AI tools trained on the best websites of the open web are now killing traffic to these sites.
And despite changing the future of journalism and our access to an online public resource of information, the issue is getting little attention.
The term Google has become so ubiquitous in our lives that it has become synonymous with the word “search”. (AP: Matt Rourke)
From search engine to answer engine
The two best examples of this trend are the choices made by Meta and Google to sideline news site content and referrals.
Just under a decade ago, Meta abruptly changed its algorithm to show less news content on the main feed, beginning a process that culminated a year ago, when it announced it no longer wanted any news content.
Now it looks like Google is doing something similar.
The facts are different, but the consequences for journalism are the same.
Google has built an AI model partly trained on news content (which it initially took for free, without asking) that’s able to answer many user queries and therefore avoid having to direct them to the sites themselves.
As a result, search traffic to news sites appears to be tanking.
This week, Google went one step further, rolling out the advanced search tool AI Mode to Australian users, which will further sideline news sites and embed the shift from “search engine” to “answer engine”.
Some publishers are discussing “Google Zero” — the hypothetical moment when Google Search stops sending any traffic to third-party websites, instead providing AI overviews directly on the search results page.
“I think it’s an absolutely pivotal, maybe the most pivotal moment for journalism ever,” Kevin Indig, a US-based search engine optimisation (SEO) expert, told me.
“A lot of publishers are asking themselves, what is our business model?”
“What is our model when AI answers so many questions?”
Newsrooms optimise for Google
It’s hard to overstate Google’s importance to online journalism.
Over the past two decades print readership declined, radio and TV audiences dwindled, and social media platforms became fickle.
Facebook turned away from news and the responsibility of moderation. Twitter got weird. YouTube was overrun with misinformation.
Through all this, Google was pretty solid. The essential utility for the web continued to direct billions of users to news sites, so ubiquitous it was almost invisible.
Optimising articles so they ranked higher in Google became second nature. News sites employed staff to spot search trends.
Google isn’t just another way of getting eyeballs on an article. Its search engine is baked into the concept of a news website.
And it goes further than this.
Given Google’s dominance of the search market, Google search is arguably central to the operation of the open web — of the hundreds of millions of publicly accessible websites that get most of their readers via Google.
A whole SEO industry exists on the basis that Google will continue to sift the web and publish search results leading users to the most relevant sites.
Another huge industry, content marketing, is based on the idea that publishing useful or generally interesting information on a topic will draw readers to a site, where they may then purchase a product or service.
As Indig put it, until recently, “every participant of the marketplace was winning.”
“Whether they’re journalists, or content marketers, or advertisers, or Google, they were all winning. It was a great deal.”
Then generative AI changed the game.
Loading
Has Google done a rug pull?
About a year ago Google rolled out generative AI answers, posted above the standard blue links, for some search queries.
Since then, news sites have reported steep declines in search traffic of 30 per cent. Some content marketers saw declines of 80 per cent.
There’s broadly two opposing arguments here. The first is that Google is giving users what they want — the internet changes and we all have to adapt. The second is that this is a rug pull.
According to the second argument, Google using AI to answer search queries isn’t just an evolution of search, but a betrayal of the basic quid pro quo.
The open web is based on the idea of a reciprocal and mutually beneficial relationship between publishers and search engines like Google.
But Google’s AI is trained on open web data, which includes news sites.
So Google is arguably using content from news sites that costs these newsrooms readers and advertising revenue. It’s gone from helpful partner to publishing rival.
Instead of Google sending readers to other sites, it’s pursuing a future where there’s only one big site: Google.
Now, maybe fears of “Google Zero” are an overreaction.
Google still sends billions of readers to news sites.
Services like Google Discover remain an important source of traffic for news.
Google needs content from news sites to train its AI, so the AI can generate answers that are factual and up to date.
It also needs news sites to do well so it can make money from selling ads on these sites.
But then again, it’s early days. We’re only one year into the experiment.
There’s widespread speculation Google will make AI Mode (a chatbot-like generative AI search tool) the default for search, which would mean less traffic to news sites, and probably mass lay-offs and fewer publishers.
And what then?
The future of journalism could be pretty bleak if publishers were only being kept alive to train AI chatbots for a US tech company.
This week Google rolled out the chatbot-like “AI Mode” for search in Australia. (Supplied: Google)
What can publishers do?
Publishers could demand licensing deals so that Google and others pay them for the content used to train AI models.
This is already happening. In August, Google signed a deal with Australian Associated Press to provide content for its AI platform.
But this approach has its drawbacks. Licensing deals favour big news sites. Smaller, independent publishers could miss out, as happened with the News Media Bargaining Code in 2021.
This inequality around licensing deals contributed to the net loss of about 166 news outlets in Australia over the past five years.
Another option is regulators like the Australian Competition and Consumer Commission could demand AI companies fairly compensate publishers for content.
Regulating US tech giants is tricky. Meta pulled Australian news from Facebook and Google initially threatened to pull its search engine rather than pay publishers for content under the News Media Bargaining Code.
A third option is an automated system for paying content owners for their work.
In July, the online infrastructure company Cloudflare announced the trial of a service that allows content owners to charge AI crawlers for access.
So far, there’s no broad consensus about what works, Indig said.
In the meantime, across the open web, gates are going up. More publishers are using paywalls to make up for lost ad revenue.
“For a lot of publishers, gating wasn’t viable,” he said.
“Now it’s like they’re being pushed towards the cliff and gating is one of the best kinds of options that they have.”
Expect to see more gated content, fewer publishers, and perhaps fewer articles overall.
From now on, you may see fewer explainers, how-to guides, or other content that’s generally more easily summarised by AI.
After this rug pull, the web is going to be less open.