Top Online Sites Debate Cutting Off Google's Crawlers (futurism.com) 47
Futurism reports:
[Some online publications] are now debating whether to cut Google off entirely, as the Wall Street Journal reports, illustrating an increasingly fraught relationship between the tech giant and the publishers that are creating content its AI models are regurgitating. According to the newspaper, prominent outlets including USA Today, Politico, the Economist, People, and Reuters are all reexamining their relationship with Google. Some are debating whether to continue to work with the tech giant at all... Even Reddit executives are reevaluating the company's $60 million-a-year contract that allows Google to train its AI models on user-submitted content on the platform. They've similarly watched as Google's AI features discourage users from navigating to Reddit...
Beyond pondering whether to cut Google off, other publishers have resorted to suing the company, accusing it of illegally rehashing their intellectual property via AI summaries. It's an extremely undesirable position for publishers. By severing ties with the search giant, they could face even steeper declines in traffic. At the same time, there's seemingly little to gain from having Google's AIs crawl their content — and in the long term, it could guarantee their destruction.
Two interesting data points from the article:
Beyond pondering whether to cut Google off, other publishers have resorted to suing the company, accusing it of illegally rehashing their intellectual property via AI summaries. It's an extremely undesirable position for publishers. By severing ties with the search giant, they could face even steeper declines in traffic. At the same time, there's seemingly little to gain from having Google's AIs crawl their content — and in the long term, it could guarantee their destruction.
Two interesting data points from the article:
- "Last month, Cloudflare CEO Matthew Prince noticed that automated bot traffic had overtaken human traffic for the first time in the internet's history."
- "USA Today has seen its traffic from US users drop by almost half over the last year."
Web Rings (Score:5, Insightful)
Re: (Score:3)
Re: (Score:2)
You're assuming public web rings.
The way things seem to be going is that sharing information in the public sphere is being abused by the social media and AI companies, both by copying without attribution or reciprocal agreements, and by slop generation which drowns the information in noise. Also known as :Tragedy Of The Commons [wikipedia.org]
If we assume private web rings are to be the future, then a lot of the above issues become moot as long as the access controls prevent the general public / AI companies from being
Re: (Score:1)
That reminds me to check out a new webring Warp Point for games (etc?)
https://hackaday.com/2026/07/0... [hackaday.com]
Re:Web Rings (Score:4, Interesting)
Web Rings were a small web method. They neither scale nor are they spam resistant. If you want to network a few "We remember the good web" nerds it works, but if you want to make new sites popular it will result in a lot of business with paid links and so on until you get them to be the same crap as the current web is. There would also be pure spam webrings and webrings that look good until enough pages participate and then replace 90% of the sites with spam.
The old web worked because it was mostly noncommercial. If you want to rescue that spirit, you need to create a noncommercial niche. And there you can also use search engines (YaCy comes to mind for that size it surely scales) and make sure there are no incentives for people to change it being noncommercial. And it is hard to keep down the incentive of making money.
less need to be part of something bigger (Score:2)
The need to be part of something bigger, a cause, a crusade, a permanent protest or whatever to call it drops off when you hit 30 years old.
Many news sites, outrage of the day blogs, and other rage bait sites cannot rely on millions of views and ad displays by producing click bait rage articles void of new facts and a hash of rage inducing stereotypes.
Re: (Score:2)
My impression is I may get more captchas than the bots.
A solution (Score:5, Interesting)
A hastily thought-up solution would be to let Google crawl headlines and the bare minimum of information. This way, Google won't be able to provide all the information, so people who are really interested will actually visit your website.
Cutting Google off entirely will destroy most of your leads.
Re:A solution (Score:4, Insightful)
This is what I think too. Basically have www.mynewssite.com and content.mynewssite.com - the latter has the full articles and a no-robots policy, the former headlines, first two paragraphs, and (legitimate) SEO keywords from the article so people doing a search will come from it. The former redirects to the latter after a few seconds so people landing on it get to the right page ultimately.
Until Google ceases being abusive, I think unfortunately we're going to have to structure all new websites this way if we're to keep search working. But what this won't do is deal with the fact Gemini is going to inject itself between users and search anyway, and so may not actually let the user see search results if it thinks it has the answer.
Re:A solution [must consider the money] (Score:3)
Much as I would like to encourage solution-oriented thinking, I don't think that approach offers much hope in these days of clickbait headlines.
On the other hand, I'm not sure it matters because the problems created by AI are getting so bad that the entire notion of websearch is failing. It's like a cesspool down there (which might remind you of old Kiyo fishing out the three one-yen notes, one of the most famous scenes in Japanese literature). It's simply that the value of the "top online sites" is too low
Re: (Score:2)
I think a better answer would be for a (publishing) industry group to put up it's own site. Block Google from every page except the entry. The purpose of the site needs to be to link to the publications of the members of the group, and to act as a filter so that those not desired can be refused access.
robots.txt was a great idea as long as nearly everyone was polite and paid attention to it, but now there appears to be the need of an enforcement mechanism, probably with graded access permissions.
Re: (Score:2)
If you want to see why this won't work think about why sites show Google content behind paywalls. They need to get their content into the index to be found, so cutting off the bot from the content costs them visitors. That's why I think they're bluffing here as well. Where do they want to get the visitors? Maybe from ChatGPT linking to them?
Re: (Score:2)
That's why it needs to be a site run by a union of publishing companies, with individual sub-sites for each of the members. This probably couldn't work for news, but most publishing is more durable. Think of it as a cross between Google and Amazon, except the corporation would not be run for profit of the corporation, but for profit of the participating companies.
Re: (Score:2)
They already fail at providing a shared flatrate. Paywalls would work much better when people would pay $10 per month and could access arbitrary articles. But still each publisher wants to have their own paywall, so that people need to decide where to subscribe. They would also depend less on Google to find paywalled articles to push people toward subscribing to them, if people could just read their article with a universal subscription that is distributed across the publishers whose articles are read.
As lo
Re: (Score:2)
Any solution that gets created will be abused by SEO (and SEO is supported and encouraged by Google). Most of us don't get much organic traffic from Google anyway.
Re: (Score:2)
Google stopped doing that (and using keywords and so on) because spammers are not honest. You can't rely one someone reporting correctly how relevant their content is, if everyone wants to be on top of the search result page.
Re: (Score:2)
using [meta] keywords. Slashdot seems to eat HTML.
Re: (Score:2)
These companies have been *deliberately* feeding Google everything on their sites for years, while putting up paywalls for regular people. They *want* Google searches to find their content when people search for any random thing within their articles, even if it's not the main subject being discussed. But they *also want* to block you from seeing the actual content unless you pay, or at the very least, sign up. They've literally been using Google to provide teases to get people to pay up.
Now their business
Re: A solution (Score:2)
Re: (Score:2)
Google down-ranks websites that show something different to its crawler and to the user. It's a classic SEO technique that they have been demoting for many years.
Re: (Score:2)
Nobody's proposing doing that.
Re: (Score:2)
IP is a farce (Score:2)
Feed it nonsense? (Score:2)
Re: (Score:2)
If Google weren't able to filter spam, you would have only spam results. Don't you think the spammer try every trick they can come up with to poison Google to rank them better?
Re: (Score:3)
That's a non-sequitur. Sponsored results are not due to Google not being able to filter spam, but due to sponsors paying for the result.
You can even see it the other way round: If spam would work, nobody would pay for the sponsored result.
The Economist? (Score:4, Interesting)
Re: (Score:2)
The problem with ownership is that you also have to absorb the losses in addition to profiting from the gains. I don't see what problem ownership solves. We already have a way to get a slice of the pie for successful corporations: corporate income tax.
Bullshit (Score:4)
Google's feature don't discourage people from using Reddit. Reddit's shitty UI keep me from using it. I WAS an OldReddit viewer but they did away with that option as well. News site like USA Today are such an ad laden crap fests that it is no wonder people use any option to NOT have to view it. Don't get me wrong, I'm not defending Gurgle, they suck and we all know it, but they are a convenient scapegoat. I get 2 papers at my house, the local Yuma Sun which is like a High School paper, and the SF Chronicle. Reporting is not what it used to be, both are opinionated crap most of the time. The best stuff for facts is always AP.
Re: (Score:2)
Old Reddit is still there, but they started requiring accounts. And I fear they will abandon it eventually. Maybe Reddit then finally dies and makes place for something better, as I think they keep old.reddit around because most high quality posters use it, while the new page is mostly the clickbaity design for people with short attention spams wanting to consume instead of creating.
Re: (Score:2)
Yeah, searching for anything in Reddit using Reddit's own search function is terrible but google can actually do the job. This baffles me, along with other basic features of reddit that have been broken or very buggy for years - in a time when people can SUPPOSEDLY use AI to code 10,000 lines a day and fix bugs like there's no tomorrow.
First off (Score:2)
Three of the five listed publications make generous use of paywalls, which I am fundamentally against, so they can go fuck themselves.
Secondly, people are discouraged from going to Reddit because most of Reddit is a cesspool of garbage and the rest just looks, and functions, like shit. Old Reddit was already bad, but the UI has only gotten worse.
Rock and a hard place (Score:2)
These companies have been using Google to tease people with search results, only to confront them with a paywall when they click the links. Now the AI crawlers have made that strategy useless. I can't say I feel bad for them. They made their own bed, now they're having to sleep in it.
If they cut Google off, they will cut themselves off. There are simply too few people who will go directly to their sites, they *need* Google for than Google needs them.
Done (Score:2)
I'm recently doing a lot of "deep searches", where one or two hits are the best you can ever hope for (obscure local history topics). I found a lot of good stuff, but did so only on DDG/Bing and Yandex. Almost all of the pages I found are not indexed by Google at all. These are high-quality original sources: the personal website of a graphic artist who did the programme guides of the event I'm researching, a newspaper biography of o