Do you develop on GitHub? You can keep using GitHub but automatically sync your GitHub releases to SourceForge quickly and easily with this tool so your projects have a backup location, and get your project in front of SourceForge's nearly 20 million monthly users. It takes less than a minute. Get new users downloading your project releases today!
geekfiend writes "Today Google updated their website to indicate over eight billion pages crawled, cached and indexed. They've also added an entry to their blog explaining that they still have tons of work to do."
This discussion has been archived.
No new comments can be posted.
Personally I find that the lack of relevant pages if the biggest problem with search engines, not the lack of pages with information. It seems I always find what I'm looking for eventually, what I need improved is the time I spend looking though spam-bomb pages before I find a page with the correct information.
These spam-pages seem to be increasing; I mean those pages with just a buch of keywords or the output of some search system.
I'm especially irritated by the increasing number of highly-ranked pages that are nothing more than another search engine's results. If Google could find some way to identify and remove these from my result set, Google's usefulness to me would increase 10 times over.
Google has a problem with this because some of those searches are actually useful.
For instance, when I search for something technical, I often run into search results from DBLP, arXiv, CiteSeer and the like -- although these are really search results within themselves, they're immensely useful to me.
Since we both effectively have a conflict of interest - Google would need to figure out a way to strike a balance.
However, results from places like Starware Search are not useful, and elevates my blood pressure with all the attempts at spamming me.
Just because I use Firefox and Adblock doesn't mean I now want to visit all possible spam sites in existence.
I don't care if Starware and friends make their money from advertising or not. The point is that Google is ALREADY a search engine, and a pretty good one at that. What is the point of returning results from another search engine, especially if the other one does not
It isn't about having a better search engine, so much as it is knowing how to use it. If you are looking for information on a recipe for oriental rice using asian spice, how would you search?
Bad search example:
oriental rice recipe asian spice
Good search example:
recipe+"oriental rice"+spice
See the difference? google tries its best to get rid of the spam pages, but it won't ever combat them all. Half of the work has to be done with you understanding the best way to describe to the search engin
Search terms: oriental rice recipe asian spice Search Results: Results 1 - 10 of about 254,000 for oriental rice recipe asian spice . (0.40 seconds) Search Effectiveness: REASONABLE. good list of relivent items matched.
Search terms: recipe+"oriental rice"+spice Search Results: Your search - recipe+"oriental rice"+spice - did not match any documents. Search Effectiveness: UTTER SHITE
The user wants SIMPLICITY. If google cannot give decent results for simple search criteria, then peopl
Erm, that's only because of the bizarre plus signs the grandparent poster put in - try this [google.com]. Note to grandparent: Just about any modern search engine assumes words not prefixed by anything are to be included in the Boolean search query. No need for +.
One thing that would really help me sometimes would be if Google allowed you to do an 'exact match' search. No, I don't mean enclosing something in double quotes, that still ignores capitalization, whitespace, and most non-letter characters. I'd like to be able to search for pages that have the EXACT string '#windows EFNET', for example, or '/usr/bin/' or whatever. '/Usr/biN' wouldn't match, and nor would '#windows^^EFNET' (where ^ is equal to a space:-) ).
I sent an e-mail to Google about this and the guy who replied didn't seem to think it was possible... anyone know if it is?
What I've read on the Google help pages seems to indicate that they don't index punctuation or capitalization. When you search for something, your string is looked for within an existing index, and appropriate reference materials are shown. Including punctuation wouldn't result in any hits within their index, meaning no results.
Now, obviously, it is theoretically possible to do just about anything. But in this case, with the architecture they have in place, anyone ever doing what you're asking would require a full-text search through their multi-TB dataset, which I suspect is highly impractical.
My point is that as I understand it, Google has coded a number of shortcut tricks which allow reasonable search times, and full-text string-exact searching would prevent them from using those shortcuts, resulting in search times they don't seem to think is reasonable.
"But in this case, with the architecture they have in place, anyone ever doing what you're asking would require a full-text search through their multi-TB dataset, which I suspect is highly impractical."
Actually, they could cut that down considerably. For example, say we were doing an exact search for '#windows EFNET' as in the original example. The first thing they could do is start with a traditional search on "#windows EFNET" [google.com]. At that point, they've cut their multi-TB dataset down to just a few megs or less of likely matches (in this case, only 10 pages matched). Then they could do a full-text check on each result, looking for an exact match and discarding all the rest.
How about a NEAR operator? Sure, AND OR NOT are nice, but my results would be a lot more relevant if I could eliminate results where the search terms appeared a thousand words apart.
The same goes for duplicate information. I don't want 200 versions of wikipedia listed when I'm looking for a specific article, nor 200 times the same man page when I'm researching something different of a unix command besides the man page of a command.
I wonder if it'll take longer to index twice as many pages? Or if they, along with this change, improved their spider and/or added hardware. Otherwise I'm not sure this change is for the better, unless you like to search for really obscure topics.
Actually no. Better search results means fewer necessary searches, which in turn will make the entire process most time effective. And anyway, you can`t just stop indexing webpages just because it might take longer to index them. You just need to improve on hardware or the technology itself.
Better search results means fewer necessary searches, which in turn will make the entire process most time effective.
Search results? Are you talking about a person searching? I was mostly concerned about how quickly Google can update their complete index now that it doubled in size. I understand for my part it might get better, as long as the index is kept up-to-date.
And anyway, you can`t just stop indexing webpages just because it might take longer to index them. You just need to improve on hardware or
What the article does not point out is why this something important. For just about forever google's store has been coverging on 2**32 documents. Some people have speculated that Google simply could not update their 100,000+ servers with a new system that allowed more. Apparently they have now done the necessary architecture changes to allow for identifying documents by 64 bit (or more identifiers) and back in the business of making their search for comprehensive.
Good timing to conincide with MSN attempt to start a new searchengine too!
"You'll note that other versions of Linux are languishing at version 6.3 or even 2.2 - only Be Dope Linux Version 27.1 with AVN (Advanced Version Numbering) brings you a version of Linux numbered at 27.1".
However, the first Google bomb mentioned in the popular press may have occurred accidentally in 1999, when users discovered that the query "more evil than Satan " returned Microsoft's home page. Now, it returns links to several news articles on the discovery.
As you see on the MSN search page, the same is happening here. I doubt they've
I don't quite believe that Google would've limited themselves that way (using 32 bit identifiers for documents) - that would've been incredibly short-sighted.
Probably not short sighted, but rather an space and cpu efficiency issue. Space - If you have 64-bit doc ids, even if you index 2^48 documents you're still wasting 16 bits per stemmed word per document. CPU - dealing with 64-bit integers on 32-bit hardware usually involves multiple loads, and decreases what can fit in the hardware data caches.
by Anonymous Coward writes:
on Thursday November 11, 2004 @07:34AM (#10786082)
For just about forever google's store has been coverging on 2**32 documents. Some people have speculated that Google simply could not update their 100,000+ servers with a new system that allowed more. Apparently they have now done the necessary architecture changes to allow for identifying documents by 64 bit (or more identifiers) and back in the business of making their search for comprehensive.
As someone who routinely follows these things, I couldn't agree more with your statement. My company operates a number of sites, and over the past 6 months, we've seen an obvious trend. Sites with, say, 5000+ pages, which used to be entirely indexed in Google, gradually had pages lost from Google. A search for site:somesite.com would return 5000 results 6 months ago. 3 or 4 months ago, the same search gave maybe 1000 results. This month maybe 500 or 600. We were definitely of the opinion that Google's index was "maxxed out" and was dropping large portions of indexed sites in favor of attempting to index new sites.
Now after seeing this story, I did a search and found literally all 5000+ pages are indexed once again. This is a huge step forward for webmasters everywhere. If your site had been slowly edged out of Google's index it's most likely back in its entirety now.
Does every minor Google or Apple related thing deserve a slashdot story? Can slashdot create a "Fanboy" section for insignificant stories advocating Google (with their software patent) and Apple (with their iTunes DRM)? That way I could filter them out more easily.
Maybe it's just me, but I'd call the doubling of information available for me to search a pretty significant improvement. Especially when the last update was only a 1b increase ("only" is a relative term, of course...).
At the same time, can Slashdot create a "Curmudgeon" section for those who like to gripe about the less than monumental significance of some story topics?
Google needs to stop obsessing about the number of indexed pages, and start concentrating on the quality. Since pagerank was switched off, 2 out of 5 searches now seem to be jammed with pages full of nothing but random words and adverts. It's even more galling when the adverts are Google Ads. Much as I love Google, they're becoming increasingly less effective as a tool.
I agree search engines are so 1990. I rely exlusively on word of mouth to find websites. If Firefox would add a button to the toolbar that said 'Cool Sites', maybe with an icon of a pair of glasses, and have the button link to a webpage with links to the latest cool sites on the net, that would certainly be the end of Google and their 8 billion pages. Pah!
That was actually how Yahoo! got started. A few of college drop-outs started making a webpage linking to their favorite sites... and their friends started going to it, and their friends' friends, and their friends' friends' friends... and then somebody offered to pay them to advertise on the site. And we ended up with this [yahoo.com].
To paraphrase Churchill, Google is the worst system devised by the wit of man, except for all the others. Where else would you go? Yahoo? Hey, how about AltaVista?
The problems faced by Google in their battle against the scumbags who would game the system are faced by every other search engine. Google, IMHO, handles them better.
at the bottom of every search results page, there is a link that says, "Dissatisfied? Help us improve". I've clicked on it once or twice, when encountering a particularly spammed keyword and they have fixed it!
My bad. I'd skimmed a few things on the web, and assumed that it had been switched off. Looks instead as though Google have changed how it works. See PageRank is dead [zawodny.com]. I need to investigate further.
Does this mean that I've been missing a huge amount of important information until now? I'd just assumed that Google covered the entire relevant web but now it seems to cover the whole same amount again. My Google alerts [googlealert.com] also seem to have started producing a lot more results which suggest that a lot of these new pages are rated quite highly. Who knows how much more quality content on the web we're just not seeing?
"Does this mean that I've been missing a huge amount of important information until now?"
Maybe the steep increase is due to all the new file formats they are indexing now. That might be useful for some people (although I sometimes find it kind of annoying that a search returns MS-Word documents).
Maybe the steep increase is due to all the new file formats they are indexing now.
The steep increase is probably due to an architecture change. Google has, for a long time, been indexing around 4 billion pages. That implies that they have been giving each page a 32 bit unique identifier, and had exhausted that id space. It would be a lot of work for them to seamlessly upgrade all their software to support a larger id, and it has taken them a long time to do so. Now that they have the large jump in page
Until today you could save your google settings [google-watch.org]
without loosing your privacy [google-watch.org].
You can still save those settings but google refuses to use them when you block their cookie. In my case I
get 10 search results although I like to receive 100. Seems that they are
making many dollars on a user's cookie, and now they are a public company my
privacy is less important than "stock holders' interests".
You can still save those settings but google refuses to use them when you block their cookie. In my case I get 10 search results although I like to receive 100.
Local tabloid Aftonbladet is running a poll on search engine use:
Google (81.4 %)
Yahoo (2.2 %)
MSN (3.8 %)
Other (11.4 %)
Don't know (1.2 %)
61730 votes so far.
I'm a little surprised, either the masses who use the "default" (MSN?) aren't bothering to answer, or google is simply very very dominant and those "default using masses" do not exist [in this country].
A lot of people have been asking what the point of the artical is, why does it matter, well possibly because Microsoft announced the launch of their search engine http://news.bbc.co.uk/1/hi/technology/4000015.stm and are claiming more pages index than google (5 billion) so google have responded by effectivly doubling their pages indexed.
Looks like they've added a gazillion LiveJournal [livejournal.com] pages to their index. I used to have a Google search box on my LJ that didn't throw up relevant results until last week or so. Now it works perfectly, just like builtin search (like what you see in MT and WordPress).
On the same day that
this story hits the BBC [bbc.co.uk]. In that
story Microsoft claim that they have
5 billion pages indexed, more than
the 4.2 billion pages indexed (at that
point) by Google. The BBC have just
updated the story with the 8bn figure.
by Anonymous Coward writes:
on Thursday November 11, 2004 @07:54AM (#10786154)
apparently my sites will never get a good ranking on google because I don't want the search engine to cache the site. So I'm using meta no-archive tags. That's the only thing I can figure out why the sites rank so poorly on google, when they come up in the top 10-20 hits on yahoo and other search engines. The keywords for the searches are valid, the sites are relevant to the keyword searches, yet the sites don't show in the top 100-300 on google.
I've avoided all the usual spam type of tags (auto refreshing, hidden text, cloaking, etc.) and the sites are legitimate and on the up and up, and yet the only page or two that google is spidering are the one or few that appear to be without the no-archive tags and possibly the revisit/expire tags.
Is google's policy, allows us to cache your site, or get penalized? Anyone else run into a similar problem or can shed some light on this? The only other thing I can think of is the robots text file, that keeps googlebot, and then other spiders through a *, from entering images directories. The spiders, including googlebot, aren't restricted from entering any other directories, they are given free reign.
Anyone else with problems with no-cache, no archive, tight revisit/expire times, or similar non-spam tags that result in penalties in google ranking?
I've been using google exclusively for a few years now. But the poor page ranking of sites on my server got me wondering about other sites that may be relevant to my own searches which may be exluded or penalized by google. So I've started using Yahoo search again, as much as I hate Yahoo (what they do with advertising to Yahoo groups and Yahoo mail is a shame). It appears that Yahoo is including better results because other sites show up with higher ranking that actually are relevant. So I've learned that Google isn't as perfect as I thought it was, which was disappointing in itself. It was easy using one search site. Now I have to use two to make sure I'm getting good results. Anyone know if there is a plugin for Firefox with both Google and Yahoo search boxes on the toolbar?
I regularly watch where my nickname, full name, parents names, etc come up in google. I've noticed in the past couple of months, my hits have DRASTICALLY reduced. They just disapeared from the database. But over the past 2 days, I've gotten notifications (thanks google alerts) about new pages being indexed and voila! They come up in a search again.
Because every blogger in the universe has added at least 3 pages since the last index. I fail to see how it is significant to me that there are now 8 billion mostly worthless sites out there. The number of actually useful sites has not gone up considerably.
A bigger index does not equal better search results, however, with the press this will generate, it will equal profits.
It would be terribly easy to get trillions of pages indexed. For instance, a site I've been working on has a public calendar system, with results fished out of a database. There are very few actual events in it at the moment, but with the 'Previous' and 'Next' links it'll run from 1970 to 2038. A naïve web-crawler would index every single month for every single year, but Google would
What Google has going for them is that they combine technical know how with marketing smarts. I still use Google as my primary search engine because it produces better results. Google understands though that, in the market at large, they need to play the numbers game. Fine they say. Within hours of the Microsoft announcement, out comes this.
Frankly, I love it any time someone can best Microsoft. The next big thing may well be consumers putting their data on servers provided by the likes of Google, Mi
Yes, this is probably a troll, but anyway... I take it you've never heard of the robots.txt file? You sound like you might want to read up on it. It's designed to help control the spidering of your pages for whatever reason, particularly cases like yours or situations where a spider would get confused and end up doing something stupid (recursive stuff, etc).
-ReK
You can rant all you want, but Google still has a fair use right to your images. They are reduced resolution images and therefor legal for non-commercial use.
Not to mention robot.txt, but that is so obvious it shouldn't need to be mention.
Well, if you know that Google is indexing your site and "stealing" your bandwidth, then you must have looked at the server logs, right? You'd see the name of the search bot is googlebot. Search for it [google.com], and you'll find that the first relevant link [google.com] explains how to prevent googlebot from accessing your site.
The logs would probably also show failed attempts to find the file/robots.txt. Similar info is gained from searching on that term [google.com] as well.
I am feeding this troll because there are people who really _do_ think like that and I wish I could yell at them to their faces:)
You put content in a place where it is publically accessible. You explicitly and proactively made that content available to everyone, including 'the average surfer' and googlebots. You took no steps to make it available only to the select few of whom you approve.
Now you are all cross and bothered because average surfers / googlebots have read / copied your content, such as it is.
The solution is to drown yourself in a bucket. I have a bucket.
Robots.txt isn't some thing that only applies to Google, it is (supposed) to be honoured by all search engines, and uses the Robots Exclusion Standard. So, when you claim these are Google's arbitary rules, you are in fact wrong. They are neither Google's nor arbitary (at least no more than any web standard).
So your clue, not so much of clue, as robots.txt doesn't fit your description.
As for why you should know about it, you are putting up a web site, it is part of running a web site. You might as well complain why you need to know about HTML, CSS or registering a domain name. Quite what coming from the UK has to do with it (something I also do), I have no idea.
"I simply do not want the average surfer to be able to visit my site, I am not interested in serving my pages to them, they simply would not appreciate or understand what it is I am showing."
Then a publicly accessable webiste is the wrong place. It is not your personal space, and it isn't private. You made it available to the world, nobody made you. To turn around and complain when (some of) the world visits it is hypocracy.
It's like putting up posters around a town, then running around complaining all these people are looking at them, won't appreciate them, and you don't want them too. It's also comes across as condescending and arrogant, which probably explains the nastiness of some of the responses.
You opted in when you put up the publicly accessable website. If all search engines had to be opt in, nobody could find anything on the web, and it would use a lot of its utility. Your assumed to want them crawling becuase the vast majority of people do, they want their site to be found. If you don't though, no problem, just use the standards for stopping searches, or password protect the site. No scandal at all, just hysterics.
Showing the low res thumbnail of your image isn't violating your copyright either. The only legitimate claim you have is the amount of time it took to remove something from the cache.
The "thieves" accusation is even more ridiculous. If you put something up on the web people can see for free, you can't complain. There are options if you want to protect it. Google doesn't claim you work as theirs (which would be 'stealing' or at least copyright violation), they help people find you public web site.
If you don't want a public website but made one, whose fault is that? If you are going to run a website and can't be bothered to find out how to do it properly, you can't blame Google.
Comment removed (Score:4, Funny)
slashdotting (Score:4, Funny)
Re:slashdotting (Score:3, Funny)
Re:slashdotting (Score:2, Redundant)
Re:Image Search (Score:3, Interesting)
For those of you that don't believe that having keywords in your URLS... just use google's own story, for example.
http://www.google.com/googleblog/2004/11/google
"Google Index Nearly Doubles" is in the url and the first header. Look at how they do thinks... and your google traffic will increase.
More pages v.s more relevant pages (Score:5, Insightful)
These spam-pages seem to be increasing; I mean those pages with just a buch of keywords or the output of some search system.
Re:More pages v.s more relevant pages (Score:5, Insightful)
Re:More pages v.s more relevant pages (Score:3, Interesting)
For instance, when I search for something technical, I often run into search results from DBLP, arXiv, CiteSeer and the like -- although these are really search results within themselves, they're immensely useful to me.
Since we both effectively have a conflict of interest - Google would need to figure out a way to strike a balance.
Re:More pages v.s more relevant pages (Score:2, Interesting)
Just because I use Firefox and Adblock doesn't mean I now want to visit all possible spam sites in existence.
I don't care if Starware and friends make their money from advertising or not. The point is that Google is ALREADY a search engine, and a pretty good one at that. What is the point of returning results from another search engine, especially if the other one does not
Re:What? (Score:2, Insightful)
Bad search example:
oriental rice recipe asian spice
Good search example:
recipe+"oriental rice"+spice
See the difference? google tries its best to get rid of the spam pages, but it won't ever combat them all. Half of the work has to be done with you understanding the best way to describe to the search engin
Re:What? (Score:3, Interesting)
Search terms: oriental rice recipe asian spice
Search Results: Results 1 - 10 of about 254,000 for oriental rice recipe asian spice . (0.40 seconds)
Search Effectiveness: REASONABLE. good list of relivent items matched.
Search terms: recipe+"oriental rice"+spice
Search Results: Your search - recipe+"oriental rice"+spice - did not match any documents.
Search Effectiveness: UTTER SHITE
The user wants SIMPLICITY. If google cannot give decent results for simple search criteria, then peopl
Re:What? (Score:5, Informative)
try +the (Score:2, Informative)
The Doctor
vs
+the doctor
Re:What? (Score:2)
http://www.google.com/help/refinesearc
There IS a + operator, and you are modded "informative"....
gus
Proximity search will help (Score:3, Insightful)
This is why I've been begging google folks to implement NEAR [pandia.com] operator!
Here is an example msn search: http://search.msn.com/results.aspx?FORM=SMCRT&q=f
Re:More pages v.s more relevant pages (Score:3, Informative)
Actually.... information IS relevant data. If it's not relevant to what you want, then it is just data...
Re:More pages v.s more relevant pages (Score:5, Interesting)
I sent an e-mail to Google about this and the guy who replied didn't seem to think it was possible... anyone know if it is?
Re:More pages v.s more relevant pages (Score:5, Insightful)
Now, obviously, it is theoretically possible to do just about anything. But in this case, with the architecture they have in place, anyone ever doing what you're asking would require a full-text search through their multi-TB dataset, which I suspect is highly impractical.
My point is that as I understand it, Google has coded a number of shortcut tricks which allow reasonable search times, and full-text string-exact searching would prevent them from using those shortcuts, resulting in search times they don't seem to think is reasonable.
Re:More pages v.s more relevant pages (Score:4, Interesting)
Actually, they could cut that down considerably. For example, say we were doing an exact search for '#windows EFNET' as in the original example. The first thing they could do is start with a traditional search on "#windows EFNET" [google.com]. At that point, they've cut their multi-TB dataset down to just a few megs or less of likely matches (in this case, only 10 pages matched). Then they could do a full-text check on each result, looking for an exact match and discarding all the rest.
Re:More pages v.s more relevant pages (Score:3, Insightful)
Re:More pages v.s more relevant pages (Score:2)
Re:More pages v.s more relevant pages (Score:2)
Re:More pages v.s more relevant pages (Score:2)
I'm all alone (Score:4, Funny)
Can't figure of I should just shoot my self or maybe just open a subscription to
Re:I'm all alone (Score:4, Funny)
Re:I'm all alone (Score:3, Informative)
8 billion pages and not a single link to my blog.
Perhaps you should just tell them where it is [google.com].
Re:I'm all alone (Score:2)
You're not alone... They don't index my site either.
Re:I'm all alone (Score:2)
Do this affect how fresh their index will be? (Score:4, Insightful)
Re:Do this affect how fresh their index will be? (Score:2)
Re:Do this affect how fresh their index will be? (Score:2)
Search results? Are you talking about a person searching? I was mostly concerned about how quickly Google can update their complete index now that it doubled in size. I understand for my part it might get better, as long as the index is kept up-to-date.
And anyway, you can`t just stop indexing webpages just because it might take longer to index them. You just need to improve on hardware or
What is new about this. (Score:4, Interesting)
What the article does not point out is why this something important. For just about forever google's store has been coverging on 2**32 documents. Some people have speculated that Google simply could not update their 100,000+ servers with a new system that allowed more. Apparently they have now done the necessary architecture changes to allow for identifying documents by 64 bit (or more identifiers) and back in the business of making their search for comprehensive.
Good timing to conincide with MSN attempt to start a new searchengine too!
Re:What is new about this. (Score:4, Interesting)
Yes, they'd better fight back, as they now have a serious competitor in MSN.
It's giving very accurate results [msn.com].
Doesn't anyone find it strange that Google gave the same top result there a while back?
MSN must be using a very similar algorithm.
Maybe a bit too similar...?
*tinfoil hat on*
Re:What is new about this. (Score:2)
Re:What is new about this. (Score:2)
"You'll note that other versions of Linux are languishing at version 6.3 or even 2.2 - only Be Dope Linux Version 27.1 with AVN (Advanced Version Numbering) brings you a version of Linux numbered at 27.1".
Re:What is new about this. (Score:3, Insightful)
It now no longer shows microsoft.com as top hit.
Haha, I guess the joke reached MS headquarters.
Re:What is new about this. (Score:2)
Or maybe all the other pages mentioning "more evil than satan himself" got higher rank anyway. The same happened to the corresponding Google query.
Wikipedia/Google bomb [wikipedia.org]:
As you see on the MSN search page, the same is happening here. I doubt they've
Still works without the quotation marks .. (Score:2)
Re:What is new about this. (Score:3, Insightful)
Re:What is new about this. (Score:3, Interesting)
Re:What is new about this. (Score:4, Interesting)
Now after seeing this story, I did a search and found literally all 5000+ pages are indexed once again. This is a huge step forward for webmasters everywhere. If your site had been slowly edged out of Google's index it's most likely back in its entirety now.
Thanks G.
great but where are the .txt and directories? (Score:3, Informative)
Re:great but where are the .txt and directories? (Score:3, Informative)
Re:great but where are the .txt and directories? (Score:2)
no update on the images (Score:3, Informative)
Google makes minor change to website - news at 11! (Score:3, Insightful)
Re:Google makes minor change to website - news at (Score:2)
Re:Google makes minor change to website - news at (Score:2)
Re:Google makes minor change to website - news at (Score:2, Funny)
At the same time, can Slashdot create a "Curmudgeon" section for those who like to gripe about the less than monumental significance of some story topics?
Turn them off then. (Score:2)
I'm moderating at the mo, and I'd have moderated you 'muppet', but I thought I'd be useful instead
J.
Re:Turn them off then. (Score:2)
J.
Re:Google makes minor change to website - news at (Score:2)
Quality - not quantity (Score:3, Insightful)
Re:Quality - not quantity (Score:3, Funny)
Re:Quality - not quantity (Score:2)
Re:Quality - not quantity (Score:2)
Re:Quality - not quantity (Score:2, Insightful)
The problems faced by Google in their battle against the scumbags who would game the system are faced by every other search engine. Google, IMHO, handles them better.
Re:Quality - not quantity (Score:2)
Re:Quality - not quantity (Score:4, Informative)
Since when is Pagerank switched off?
Re:Quality - not quantity (Score:4, Interesting)
And I for one welcome... (Score:2, Funny)
Mhm to anonymous coward or not to anonymous coward?
Will moderators smack my karma below zero?
Nonsense. (Score:2, Funny)
You don't just go from 4 billion to 8 billion overnight.
They are probably just crawling the same 4 billion twice.
Re:RTFA (Score:2)
Makes you wonder... (Score:5, Insightful)
Re:Makes you wonder... (Score:5, Interesting)
Maybe the steep increase is due to all the new file formats they are indexing now. That might be useful for some people (although I sometimes find it kind of annoying that a search returns MS-Word documents).
Re:Makes you wonder... (Score:3, Informative)
The steep increase is probably due to an architecture change. Google has, for a long time, been indexing around 4 billion pages. That implies that they have been giving each page a 32 bit unique identifier, and had exhausted that id space. It would be a lot of work for them to seamlessly upgrade all their software to support a larger id, and it has taken them a long time to do so. Now that they have the large jump in page
Advertising a deficiency (Score:2)
"Oh, if you just added several billion pages, were you giving me crap before? How many more billions of pages are you not indexing right now?"
Google's announcement merely gives its users reason to question the size and comprehensiveness of Google's index.
Google needs your cookie badly (Score:2, Informative)
Re:Google needs your cookie badly (Score:3, Informative)
Create a keyword bookmark [mozilla.org] with the URL
Give it the keyword 100, then type 100 search_term in the address bar to use it.
Google domination. (Score:2, Informative)
Local tabloid Aftonbladet is running a poll on search engine use:
61730 votes so far.
I'm a little surprised, either the masses who use the "default" (MSN?) aren't bothering to answer, or google is simply very very dominant and those "default using masses" do not exist [in this country].
Re:Google domination. (Score:3, Insightful)
I think it is more that many users of IE just do not twig that their failed page access resulted in an automatic query to MSN.
In reality, most users make occasional deliberate queries to Google and more frequent accidental queries to MSN.
If I kept eating so much spam... (Score:2, Funny)
Microsoft (Score:4, Interesting)
8 billions.... (Score:2, Funny)
and 19% is pr0n.
There's debate if the remaining 1% contains pirated music and movie or plans for DIY nukes.
Mine is bigger than yours!!! (Score:5, Informative)
In a statement Microsoft said its search engine returned results from five billion web pages - more than any other search engine.
But this quickly won a response from Google which announced that its index has now grown to more than 8 billion pages.
Prior to the Microsoft announcement, Google was only indexing 4,285,199,774 web pages.
Steve Ballmer is soon to announce that his daddy is one hundrad years old, and kan kick your daddy's ass...
Grrrrr (Score:4, Funny)
Now it's going to be even harder to get my name in the top spot. Why was I cursed with the surname Smith!
Searching LiveJournal.com (Score:5, Informative)
Looks like they've added a gazillion LiveJournal [livejournal.com] pages to their index. I used to have a Google search box on my LJ that didn't throw up relevant results until last week or so. Now it works perfectly, just like builtin search (like what you see in MT and WordPress).
Doubled? Wait a minute... (Score:5, Funny)
Competing with Microsoft's 5bn? (Score:5, Informative)
I smell competition!
Rich.
Does this mean...? (Score:4, Insightful)
Re:Does this mean...? (Score:2)
meta-no-archive (Score:3, Interesting)
I've avoided all the usual spam type of tags (auto refreshing, hidden text, cloaking, etc.) and the sites are legitimate and on the up and up, and yet the only page or two that google is spidering are the one or few that appear to be without the no-archive tags and possibly the revisit/expire tags.
Is google's policy, allows us to cache your site, or get penalized? Anyone else run into a similar problem or can shed some light on this? The only other thing I can think of is the robots text file, that keeps googlebot, and then other spiders through a *, from entering images directories. The spiders, including googlebot, aren't restricted from entering any other directories, they are given free reign.
Anyone else with problems with no-cache, no archive, tight revisit/expire times, or similar non-spam tags that result in penalties in google ranking?
I've been using google exclusively for a few years now. But the poor page ranking of sites on my server got me wondering about other sites that may be relevant to my own searches which may be exluded or penalized by google. So I've started using Yahoo search again, as much as I hate Yahoo (what they do with advertising to Yahoo groups and Yahoo mail is a shame). It appears that Yahoo is including better results because other sites show up with higher ranking that actually are relevant. So I've learned that Google isn't as perfect as I thought it was, which was disappointing in itself. It was easy using one search site. Now I have to use two to make sure I'm getting good results. Anyone know if there is a plugin for Firefox with both Google and Yahoo search boxes on the toolbar?
Re:meta-no-archive (Score:3, Informative)
Closer to re-adding entries (Score:2)
Just tried the beta of the new MSN Search (Score:4, Funny)
It is not clear to me how I can help them improve. Suggest they switch their servers to Linux?
I know why it has doubled... (Score:3, Interesting)
Re:This is news ? (Score:2, Funny)
Google is a constant source of information and a geeks friend - if the index has doubled so has our supply of information. Information rules!
Re:This is news ? (Score:2)
Re:This is news ? (Score:3, Insightful)
Yeah, but it'd be news if the sun set twice in one night or rose twice as bright.
It's more the exponential increase in the size of the index rather than the piecemeal addition.
Re:This is news ? (Score:3, Interesting)
Re:This is news ? (Score:3, Interesting)
It would be terribly easy to get trillions of pages indexed. For instance, a site I've been working on has a public calendar system, with results fished out of a database. There are very few actual events in it at the moment, but with the 'Previous' and 'Next' links it'll run from 1970 to 2038. A naïve web-crawler would index every single month for every single year, but Google would
Geeks who understand marketing (Score:2)
Frankly, I love it any time someone can best Microsoft. The next big thing may well be consumers putting their data on servers provided by the likes of Google, Mi
Re:Google Schmoogle (Score:2, Funny)
Re:Google thieves my bandwidth (Score:5, Informative)
robots.txt (Score:4, Informative)
-ReK
Re:Google thieves my bandwidth (Score:2, Insightful)
Not to mention robot.txt, but that is so obvious it shouldn't need to be mention.
Re:Google thieves my bandwidth (Score:5, Informative)
The logs would probably also show failed attempts to find the file
So, to sum up... (Score:5, Insightful)
I am feeding this troll because there are people who really _do_ think like that and I wish I could yell at them to their faces
You put content in a place where it is publically accessible. You explicitly and proactively made that content available to everyone, including 'the average surfer' and googlebots. You took no steps to make it available only to the select few of whom you approve.
Now you are all cross and bothered because average surfers / googlebots have read / copied your content, such as it is.
The solution is to drown yourself in a bucket. I have a bucket.
Re:Read more carefully. (Score:5, Insightful)
Robots.txt isn't some thing that only applies to Google, it is (supposed) to be honoured by all search engines, and uses the Robots Exclusion Standard. So, when you claim these are Google's arbitary rules, you are in fact wrong. They are neither Google's nor arbitary (at least no more than any web standard).
So your clue, not so much of clue, as robots.txt doesn't fit your description.
As for why you should know about it, you are putting up a web site, it is part of running a web site. You might as well complain why you need to know about HTML, CSS or registering a domain name. Quite what coming from the UK has to do with it (something I also do), I have no idea.
"I simply do not want the average surfer to be able to visit my site, I am not interested in serving my pages to them, they simply would not appreciate or understand what it is I am showing."
Then a publicly accessable webiste is the wrong place. It is not your personal space, and it isn't private. You made it available to the world, nobody made you. To turn around and complain when (some of) the world visits it is hypocracy.
It's like putting up posters around a town, then running around complaining all these people are looking at them, won't appreciate them, and you don't want them too. It's also comes across as condescending and arrogant, which probably explains the nastiness of some of the responses.
You opted in when you put up the publicly accessable website. If all search engines had to be opt in, nobody could find anything on the web, and it would use a lot of its utility. Your assumed to want them crawling becuase the vast majority of people do, they want their site to be found. If you don't though, no problem, just use the standards for stopping searches, or password protect the site. No scandal at all, just hysterics.
Showing the low res thumbnail of your image isn't violating your copyright either. The only legitimate claim you have is the amount of time it took to remove something from the cache.
The "thieves" accusation is even more ridiculous. If you put something up on the web people can see for free, you can't complain. There are options if you want to protect it. Google doesn't claim you work as theirs (which would be 'stealing' or at least copyright violation), they help people find you public web site.
If you don't want a public website but made one, whose fault is that? If you are going to run a website and can't be bothered to find out how to do it properly, you can't blame Google.
Re:Ofcourse ... (Score:2)