Google Now Defaults To Not Indexing Your Content

An anonymous reader a report:Google is no longer trying to index the entire web. In fact, it's become extremely selective, refusing to index most content. This isn't about content creators failing to meet some arbitrary standard of quality. Rather, it's a fundamental change in how Google approaches its role as a search engine.

From my experience, Google now seems to operate on a "default to not index" basis. It only includes content in its index when it perceives a genuine need. This decision appears to be based on various factors:
Extreme content uniqueness: It's not enough to write about something that isn't extensively covered. Google seems to require content to be genuinely novel or fill a significant gap in its index.
Perceived authority: Sites that Google considers highly authoritative in their niche may have more content indexed, but even then, it's not guaranteed.
Brand recognition: Well-known brands often see most of their content indexed, while small or unknown bloggers face much stricter selectivity.
Temporary indexing and de-indexing: In practice, Google often indexes new content quite quickly, likely to avoid missing out on breaking news or important updates. Soon after, Google may de-index the content, and it remains de-indexed thereafter. So getting initially indexed isn't necessarily a sign that Google considers your content valuable.

Coming soon

[93 Escort Wagon]

- Google announces a program where site owners can pay a fee that indicates their website should be considered highly authoritative and unique.

the search engine that Gotham deserves

[Pseudonymous Powers]

You either die an Alta Vista or live long enough to see yourself become Yahoo.

Re:

[Anonymous Brave Guy]

I suspect in this case, you either die a best-in-kind search engine or live long enough to see a combination of SEO gaming and generative AI render your product obsolete.

Let me know when someone figures out how to do seamless product placement within LLM-generated answers and I'll show you the next Google.

Re:

[spiritplumber]

I have that, how do I monetize it?

Re:

[Narcocide]

Get bought by Google.

Re:

[ArchieBunker]

Could we resurrect Altavista? It was awesome to actually find what you're looking for. I've been playing around on the internet for nearly 30 years now. Not once have I ever used Yahoo to look for anything. On occasion I would play billiards with someone or use the chat program but never the search.

Re:

[Waffle Iron]

That's not how I recall it. Altavista seemed to have a very rudimentary algorithm based on literal text matching, and crafting queries to get what you wanted was often tough. They had also started piling on annoying features to monetize their site.

I still remember when Google showed up: Back then it returned far superior results quickly and without clutter (I'm not claiming that this is still the case, BTW). I and most everyone I knew switched to exclusively using Google almost instantly.

Re:the search engine that Gotham deserves

[_merlin]

Google results did suck in 2010. It was the end of the '90s to the mid '00s when Google was "good".

Re:

[timeOday]

In 1940 the A6M Zero fighter plane had a 12:1 kill ratio in Japan's favor. By 1945 it was obsolete except for kamikaze attacks. That's the history of AltaVista between 1996 and 2000.

Re-launching AltaVista on the modern web would be like sending the Zero up against an F35.

Re:

[excelsior_gr]

For a rather short period of time, metasearching became popular and I remember using Copernic (which, ahem, Google tells me is still around as a desktop search tool). It would show you the results and would also indicate the engine that delivered them. At some point you would realize that Google dominated the top results and that's when people stopped bothering with the other (meta)engines and would just go to Google. Google also had a front page with no clutter (it actually still does!) that loaded quickly

Re:

[ixuzus]

I'm not convinced on the superior results. I circles I moved in the reason for the switch was much simpler that that. The Google landing page was fairly small and loaded quickly. I know a lot of people who switched to it because they didn't have to wait for the news, sport, weather, shopping, and other random stuff to load before they could search. In days when there was still a lot of dialup that made quite a big difference..

Re: the search engine that Gotham deserves

[jovius]

The best modern analogs for Altavista are general-type dark web search engines. They are not particularly effective or useful. A funny alternative to Altavista was the warez search site Astalavista. A pretty sure way to get infected too.

Google was next level and changed the game completely. The early years of Google with dead simple usability and magic-like functionality would be nice, so somewhere in the late 90s to 00s.

Re:

[NoWayNoShapeNoForm]

- Google announces a program where site owners can pay a fee that indicates their website should be considered highly authoritative and unique.

That simply dovetails with any business practice that they MAY HAVE for site owners to pay Google to improve their page ranking or even pop to the top.

Re:

[strikethree]

- Google announces a program where site owners can pay a fee that indicates their website should be considered highly authoritative and unique.

The absolute Truth of your statement hit me like a ton of bricks. In my mild disorientation, while "coming back to Reality", my mind was open enough to see a parallel:

Ever watch a child grow up from infancy to death? When they were a baby, they uncritically took EVERYTHING in as Truth. Then, they run across a lie and become more circumspect in what they ingest. Eventually, they become bitter and the only things you can get them to ingest are things that directly benefit them. Eventually, they become resista

Their only option

[migos]

SEO is ruining indexed search. A lot of times when I need answers I prefer ChatGPT. I would only google if my goal is to end up on some website. As for new websites, they'll just have to promote via social media or create unique content.

Re:Their only option

[Anonymous Coward]

You must be asking pretty dumb questions if "chatgpt" can answer them.

Re:

[Anonymous Coward]

I don't think it is either, I think they're trying to increase engagement on the actual search engine itself, which comes at the expense of relevance.

Re:

[vivian]

I also find ChatGPT more useful for finding answers to things - because I get an answer directly back with usually fairly relevant information, instead of a page of summaries to websites that have been padded out massively with someone's life story about the subject in an attempt to get ranked.
For example, it used to be simple to get say, a generic muffin recipe, using Google. Now the results that come back are full of fluff, probably a link to a video on youtube and if you are lucky, the actual ingredients

Makes sense

[vakuona]

Google is not your memory. It is a service to find information that you are, or may be interested in.

If this change means that Google starts to rid its index of content farms and other not particularly useful sites, then it can only make Google better.

Also a shot across the bow for anyone who wants to demand money from Google for indexing their website.

Re:Makes sense

[Luckyo]

Problem here is that Google has become "the web" for most people. It's a de facto monopoly in web search, which makes it a de facto gatekeeper of the web.

That means that as it moves to be more selective, it effectively renders freedom of expression that enabled internet as a revolution of being able to find so much niche information into a cable TV-style highly selected, highly curated and highly constrained source of knowledge.

The question then becomes, are people going to switch? Or are LLMs replacing web search entirely anyway?

Re:

[GoTeam]

... It's a de facto monopoly in web search, which makes it a de facto gatekeeper of the web.

No! Bing is every bit as capable as Google!!! /s

Re:Makes sense

[Luckyo]

Hilariously, my search kit includes three things.

Startpage to give me Google results. It's generally best for finding things overall.
DDG to give me Bing results. It tends to be better than Google in certain specialist things but worse overall.
Yandex to find things that are censored on the other two. It's worse than other two for everything else.

Problem is that most people are not this meticulous in searching for something, because they have better things to do than hunt a specific thing. A general answer is good enough in most cases for most people. This is why Google changing in the way it's being suggested is going to cause problems.

Re:

[SPQRBob]

Or when you can't find shady enough porn elsewhere and Tor scares you.

Re:

[Vlad_the_Inhaler]

What I'm finding more frequently nowadays is that DuckDuckGo is returning msn.com results for "current event" queries. Since I do a lot of my browsing using a little known browser in Private Mode - and msn.com can't handle that combination - DDG is becoming less useable for me. Independently of that problem, DDG (says) it does not track its users' queries, but when the results point to msn.com that non-tracking starts to become pretty useless.

Re:

[hdyoung]

The initial revolution of the internet as total-free-for-all was NEVER going to last. Set aside all the financial incentives to control the internet. It's just plain old tragedy of the commons. Originally, every internet page was something that was coded or at least designed by a human with something to add. It made sense to catalog everything. But, it's gotten literally TOO EASY to publish content on the web. Nowadays, every company, government agency, and schmoe with an axe to grind can basically do this:

Re:

[Luckyo]

Let me correct the actual concern:

10 generate malinformation in web page format

20 publish on web

30 goto 10

Remember, there are three things of concern on the internet according to DHS report that defined those. And the main concern isn't misinformation, but malinformation. That is defined as factually correct information that is against the mainstream narrative.

Let me give you an example of such malinformation, so you understand why it's by far the worst one.

"If we didn't count African American violent crime

Re:

[Savage-Rabbit]

Google is not your memory. It is a service to find information that you are, or may be interested in.

If this change means that Google starts to rid its index of content farms and other not particularly useful sites, then it can only make Google better.

Also a shot across the bow for anyone who wants to demand money from Google for indexing their website.

If it's only link farms and the likes I'm OK with this change as long as that's all it is and not completely destroying the ability to search on obscure subjects.The most valuable searches are generally the esoteric ones. There is a whole string of search engines that I stopped using because Google gave me good results on all manner of esoteric searches but the other search engines didn't. I don't really care if a search engine gives me good results on: "Taylor Swift", what I value is good results on esoter

Re:Makes sense

[tlhIngan]

If this change means that Google starts to rid its index of content farms and other not particularly useful sites, then it can only make Google better.

Or the current bane, content farms that are AI generated.

Content farms are bad enough, but with AI, they are exponentially worse .

Re:

[Morromist]

You're right but i think there's also a pushback. People don't like being shoved into the basement while there's a party upstairs. That's why I left twitter, for example. Now in 1 year threads is already half the size of twitter. Not that threads isn't shoving people into the outer darkness too, but they aren't as obvious about it, demanding money for the privlage of being heard.

Contents Page not Index

[Roger W Moore]

It sounds like Google is changing from being the index of the internet to simply being the contents page. I can see why since I'm sure the vast majority of searches and hence ad revenue are for common topics and news-related content but I always liked the fact that they used to index pretty much everything so that, no matter how obscure your search you could almost always find something relevant.

Re:Contents Page not Index

[Seven Spirals]

Stopped using Google in 2021 due to censorship. I've noticed they've gone rapidly downhill since that point. Their primary problem seems to be altering searches for SEO slime. Their secondary and tertiary problems are political censorship and filtering/bad-results at a level that weeds out the actual results one wants. When shit-search-engines like Yandex are giving me better results, all it takes is a few competitive searches to see the jig is up on the quality. Knowing they are censorious fuckwads simply makes me celebrate leaving for greener pastures years ago. Google delivers garbage search and garbage software (until they revoke or withdraw it like they always do).

Re:

[Seven Spirals]

how did you notice

I guess you didn't read this part "all it takes is a few competitive searches". So, in addition to being a coward, you're a moron, too.

Re:

[Malay2bowman]

Get woke go broke. Good luck posting comments on YouTube videos these days because it's so fucking easy to unintentionally "offend" the AI mod bot, and the narrative is carefully and strictly controlled there. And then it's shadowban time.

Who is the author?

[bad-badtz-maru]

This is just some rando dude's thoughts with nothing supporting it. It may be true, may not be true, who can tell?

Re:

[bad-badtz-maru]

This is just some guy's thoughts. There is no official statement, in the article, from Google. This is all happening on the internet.

Re: Who is the author?

[Doctor O]

10 years experience and entrepreneur, in building applications as a freelancer, no less.

Cute.

Re:

[jhecht]

Just what I've been wondering. Here's what Google says. https://developers.google.com/... [google.com]

Re:

[bad-badtz-maru]

See what Google says for "rhetorical question".

Google continues to shoot own foot with Uzi

[Malay2bowman]

Meanwhile I am using Qwant.com for stuff that is more than just looking up a word and it's meaning. But Google will fuck this one up too, with AI and synergy, and whatever glue of choice they are huffing.

Re:

[Malay2bowman]

"Quick looking up" a word", which Goog... Sear... is about only good for these days. But I'll have to use Qwant for this too sooner or later as Google continues it's quest to snatch defeat out of the jaws of victory.

if websites want to be indexed

[FudRucker]

by search engines then maybe make an "index.txt" or "index.html" for search engines to easily scoop up and include in their search

Noble goal, but...

[MpVpRb]

I love the idea of eliminating AI crap and SEO
Unfortunately, this will hurt a lot of small creators like me who have a real small business
It's kinda like wikipedia that deletes useful stuff because they judge that it's "not notable"
Google has been getting increasingly useless. This change will eventually mean that the only search results that are presented are ads and content from the famous

No longer a search engine

[Culture20]

More like a search feature in America On Line. You can search for stuff inside the AOL environment, but only what AOL deems worthwhile to put therein.

Re:

[AsmCoder8088]

While most people think of AOL as a walled garden (and understandably so), it had its roots in the Quantum Link (QLink) days of the Commodore era and in fact a lot of the AOL communication protocol was derived from that (FDO tokens and so forth). Once you got past the "Welcome" screen there was a lot you could do, and it enabled a lot of small-time software developers to host their shareware programs for others to try out. I remember getting a lot of early Macintosh games that way (TaskMaker, for example)

Google is an advertising company

[TomGreenhaw]

It's looking like if you want to be found in a google search, you either have to be very lucky or an advertiser that pays them to include you in results.

Google

[Anonymous Coward]

Google is a reddit search engine with ads

What I think this translates to:

[Rick Schumann]

"We only want you to see the sites and content we want you to see"

Time to stop using Google

[gweihir]

They have now completely gone off the rails.

google needs to take a few steps back

[Jayhawk0123]

google needs to take a few steps back and reverse the changes they have done in the last few years to their search algorithm. The search result accuracy has been steadily declining. With results being either dated, or completely out of context from what the search was for.

Is this new?

[Touvan]

Around a decade ago I used to play with things on my personal blog just to see how it would impact search rankings, and traffic from google. One thing I noticed as a very clear trend, was that if I had Google Analytics on my blog, and/or Google Ads, I'd get a lot better SEO. I never believed that was coincidence. Google has ALWAYS been a business which sells its users to advertisers.

Just the latest fundamental change in Google

[Mononymous]

Originally, Google was a search engine. It crawled the web and generated an index.
People who were using the web knew how to type, and a little bit of how computers work.
They typed in text to search for, and Google returned links to pages containing that text.
But over the following Septembers, the behavior of the users changed.
The people at Google realized that more and more people weren't using the search the way it was written. A lot of them were asking questions, trying to use Google to get information, instead of to look for web pages.
So the behavior of Google changed, both to give those people more useful results, and to give Google more opportunities to sell advertising.
That's why Google hasn't worked as a straight search engine for a long time. Oh, how I miss that search engine.
Since it's not a search engine anymore, indexing the whole web no longer makes sense for Google.

Re:

[swillden]

That's why Google hasn't worked as a straight search engine for a long time. Oh, how I miss that search engine.

Whenever this topic comes up, I see a lot of comments like this. If it's really possible, and desirable, to have a search engine in the style of Google circa 2000, why isn't anyone running one? Surely it would be wildly successful?

What I see out there is Bing, which does all the same things Google does, but not quite as well, and DDG, which is Bing plus a privacy shield, which makes it a bit worse than Bing -- but still very much trying to compete directly with Google as it is now, rather than offering

Article written by an SEO guy

[Tony Isaac]

Right there at the very top of the headline in the linked article, "SEO."

This guy is upset because his SEO tactics aren't working as well any more.

The only ones scummier than Google, are the SEO companies. I'm glad Google seems to be winning that battle.

No actual knowledge, just a guy's observations

[Tony Isaac]

This guy is complaining because HIS sites aren't getting indexed as quickly as they used to. Maybe his content is just SEO BS, a type of website Google explicitly combats.

I personally run a genealogy website that has several hundred active users. I didn't have to do a thing to get Google to index it, they just picked it up on their own. But then, this site has actual content that people want to find, not made-up content that an SEO company wants you to find, instead of what you're actually looking for.

Re:

[ZiggyZiggyZig]

I kind of agree with you, but my own personal experience matches his, unfortunately. I revamped my personal website at the end of November 2022, and since then, none of my pages have been reindexed - and some of the pages of the old website have been deindexed, even with the 301 redirects I explicitly put there for helping people accessing the website.

At some point I thought that was because I didn't put the Google Analytics spy in my pages and did not register to the Webmaster Tools, for ethical reasons. I

Sucks to be a webcrawler

[Gabest]

Most websites are behind a cloud hosted "I'm not a robot" captcha, and all the content is javascript generated.

You have to jump through their hoops now.

[jonadab]

Yeah, among other things, you now have to go onto Google Maps, find your location, and go through the "claim this business" process, even if you are not in any way shape or form a business. As part of my job, I run the website for a public library. A few months ago, we suddenly went from being pretty much the entire first page of results (if someone search for a library in our town), to not showing up at all. Once we claimed the Google Maps listing and put our URL into the "website" box on its profile, w