Forgot your password?
typodupeerror
Google Businesses The Internet

Google URL Index Hits 1 Trillion 249

Posted by Soulskill
from the orders-of-magnitude dept.
mytrip points out news that Google's index of unique URLs has reached a milestone: one trillion. Google's blog provides some more information, noting, "The first Google index in 1998 already had 26 million pages, and by 2000 the Google index reached the one billion mark. Over the last eight years, we've seen a lot of big numbers about how much content is really out there. To keep up with this volume of information, our systems have come a long way since the first set of web data Google processed to answer queries. Back then, we did everything in batches: one workstation could compute the PageRank graph on 26 million pages in a couple of hours, and that set of pages would be used as Google's index for a fixed period of time. Today, Google downloads the web continuously, collecting updated page information and re-processing the entire web-link graph several times per day."
This discussion has been archived. No new comments can be posted.

Google URL Index Hits 1 Trillion

Comments Filter:
  • Screenshot. (Score:5, Funny)

    by Shaitan Apistos (1104613) on Saturday July 26, 2008 @12:05AM (#24345505)

    Or it didn't happen.

  • by loconet (415875) on Saturday July 26, 2008 @12:09AM (#24345527) Homepage

    Once the index reaches a google (or rather a googol), the universe explodes.

    • by txoof (553270) on Saturday July 26, 2008 @12:14AM (#24345551) Homepage
      Is that the modern equivalent of the Mayan calendar running out of days?
    • Re: (Score:2, Insightful)

      by Anonymous Coward

      Won't happen since the universe's max integer is significantly smaller than a googol or a 'google'.

    • Re:How long till.. (Score:5, Insightful)

      by rho (6063) on Saturday July 26, 2008 @01:38AM (#24345917) Homepage Journal

      I'm more interested in when Google starts returning relevant results to my queries.

      I can't believe that I'm the only one that finds Google's quality of service somewhat below par. I guess they're better than randomly stabbing in the dark, and there certainly isn't any alternative that's obviously better, but Google sure isn't everything they think they are.

      I know--stop trying to compete with Wikipedia and cut out Experts-Exchange.com from your search results since their pages don't actually return the information you think they do.

      • Re:How long till.. (Score:5, Informative)

        by onedotzero (926558) on Saturday July 26, 2008 @02:05AM (#24346013) Homepage

        ... and cut out Experts-Exchange.com from your search results since their pages don't actually return the information you think they do.

        Perhaps you should try scrolling to the bottom of the page... :)

      • Re:How long till.. (Score:5, Informative)

        by cdrudge (68377) on Saturday July 26, 2008 @02:05AM (#24346015) Homepage

        It took me a while to realize it, but if you scroll clear to the bottom of an expert exchange post, you'll find the comments unhidden and relevant.

      • Re:How long till.. (Score:5, Informative)

        by Anonymous Coward on Saturday July 26, 2008 @02:07AM (#24346021)

        ...and cut out Experts-Exchange.com from your search results since their pages don't actually return the information you think they do.

        If you block cookies from experts-exchange.com you can actually see the answers on any e-e page - after you visit the first time, it normally sets a cookie to not show results next visit, which is how they get Google to index their pages anyway. With cookies from them blocked, you can then see the answers - you just have to scroll 7/8s of the way down the page past all the fake "Please sign up to see this result" boxes.
        (First AC post in years... tee hee. :)

        • by fyoder (857358) *

          If you block cookies from experts-exchange.com you can actually see the answers on any e-e page

          I don't know if Americans would be allowed to do that. Sounds like a potential violation of the DMCA.

      • Re:How long till.. (Score:5, Interesting)

        by blahplusplus (757119) on Saturday July 26, 2008 @02:22AM (#24346097)

        "I'm more interested in when Google starts returning relevant results to my queries.

        I can't believe that I'm the only one that finds Google's quality of service somewhat below par."

        You're not the only one, but for the most part it is better then most other search engines out there. The real problem is spammers and paid advertising, I think spammers have really made search frustrating for a lot of companies. And ad companies pay other people to promote their sites for them (digg, slashdot, etc). I've noticed the increase in spam-vertised websites in search results for a lot of things.

        Personally I think the idea of sharding and search being more specific for what you're looking for is needed. I'd like to see a google with 'tags' and a delicious interface, things like educational institutions and universities get lumped into their own search engine space for instance, this would help narrow down what one is looking for, although it would take time and feedback to design something well for other areas. The fact is that search results get diluted as you put more and more stuff online (numbers and geometric scale).

        For fun, I've noticed stumble upon and del.ico.us are not bad alternatives when looking for new and interesting sites without having to use search

        • by peragrin (659227)

          you do know you can use google to search specific websites right? So if your looking through a specific domain you can add that to your search query.

          The other trick is to add search terms until you have gotten a smaller list of results.

        • Re: (Score:2, Interesting)

          by MPAB (1074440)

          Try searching for a given product and the word "review" or something alike. You'll get endless pages of stores with no review whatsoever that must be scrolled away till you find a real technology site that has actually reviewed the product.

      • by Mike89 (1006497)
        If they cut out all the stupid "Find it cheapest here!!" style domains, I'd find it much more useful.
      • Re:How long till.. (Score:4, Interesting)

        by hairyfeet (841228) <bassbeast1968@@@gmail...com> on Saturday July 26, 2008 @06:12AM (#24346801) Journal
        That is why I switched to Yahoo in FF. Google search just seems to be getting crappier of late while Yahoo seems IMHO to be getting better. At least fo me it hits what I'm looking for on the first page a lot more than Google,and I love when i type something like Bioshock I can hit the more tab and get Bioshock demo,cheats,patches,reviews,etc. Whereas you hit the more tab in Google you just get more crap like Google groups and blogger. But of course it will probably be bought out by MSFT and turn into a giant turd like Live Search,so i'm just going to enjoy it while it lasts. And as always this is my 02c,YMMV
      • Re: (Score:2, Interesting)

        I'm more interested in when Google starts returning relevant results to my queries.

        I can't believe that I'm the only one that finds Google's quality of service somewhat below par. I guess they're better than randomly stabbing in the dark, and there certainly isn't any alternative that's obviously better, but Google sure isn't everything they think they are.

        I find this larger index rather unsettling as I feel my search results are becomming more unrelevant. Mostly due to the following reasoning.

        Finding 1 page in a billion page index is relatively easier than finding 1 page in a trillion page index.

        Has the relevance of the results kept up with the growing index size? or does the growing index cause more unrelevant pages to appear higher in the search results?

        In my opinion the next big advancement that is needed on the web is better auto-catagorisation of conten

    • Re: (Score:2, Funny)

      by bliip (1104227)
      Slightly less catastrophic is the result of typing "google" into google: http://uk.youtube.com/watch?v=Fet0SCt7uGg [youtube.com]
  • Amazing (Score:3, Interesting)

    by SoupIsGoodFood_42 (521389) on Saturday July 26, 2008 @12:13AM (#24345541)

    As someone who is partially engineering/analytically minded (but not a great programmer) it amazes me how Google has manged to index so much data, yet at the same time, serve up results in a fraction of a second to so many people.

    • Re:Amazing (Score:5, Insightful)

      by timmarhy (659436) on Saturday July 26, 2008 @12:20AM (#24345581)
      i wish they would work on weeding out the crap. anything you google now is infested with cheesy search sites that list other websites and try plaster you with ads. they contribute zero to the web.
      • Re:Amazing (Score:5, Informative)

        by Freaky Spook (811861) on Saturday July 26, 2008 @12:39AM (#24345691)

        I couldn't agree more.

        Many of the clients I support are constantly asking me "Is there a program that does this? or Can you find me a program to do this" etc etc.

        I used to be able to just use google to help me get started but these days the top level searches are all those bloody link farms peddling "free" software, even when typing in the word review you come up with link farms that offer no reviews.

        • Re:Amazing (Score:5, Informative)

          by arotenbe (1203922) on Saturday July 26, 2008 @01:27AM (#24345877) Journal

          Many of the clients I support are constantly asking me "Is there a program that does this? or Can you find me a program to do this" etc etc.

          I used to be able to just use google to help me get started but these days the top level searches are all those bloody link farms peddling "free" software

          Have you tried SourceForge [sourceforge.net]? That's what it's there for, you know.

        • Re: (Score:3, Interesting)

          by hairyfeet (841228)
          Just go to Freewareworld Team [all4you.dk]. I work in PC repair and am always needing a piece of freeware to do a specific task. Freewareworld Team has an excellent search engine. Simply type what you need the program to do and they find you a freeware program that does it. And I have never run into spyware or trialware there. So enjoy and I hope this helps.
        • Re: (Score:3, Insightful)

          by cheater512 (783349)

          If you design your queries well enough, then you dont see any of the crap.

      • Re: (Score:3, Interesting)

        Yeah, that's a problem. I'm sure they'll work it out. I don't find it to be a problem most of the time though, just on certain searches in certain places. They have a real spam problem if you search for info on pharmaceuticals in their groups search last time I checked (about a month ago). The problem wasn't the Usenet groups, but their own special groups, and the worse thing is you can't filter out their groups and just search Usenet ones.

        I tried to contact them about it and discovered that they could also

      • Try "Live" search (Score:3, Interesting)

        by symbolset (646467) *

        And you'll be back faster than a Google search result. Weeding out the crap?

        Just for a sample, try this one: getfirefox [live.com]. If the first link on that search goes to a Mozilla mirror you will win one Internet. Try Linux [live.com]. Hey, this is fun. Spoiler: the first link there is always "www.Microsoft.com/Windows : Special Offers from Windows Vista® w/ the Purchase of Select Laptops." The first time I tried this I was looking for Open Office and wound up misdirected to a members only site where you had

        • by pagaboy (1029878) on Saturday July 26, 2008 @04:12AM (#24346431)
          Turns out Live.com's market share for today has tripled due to Slashdot users clicking on the above links...
        • All three of his examples go directly to the most official site for Firefox, Linux and OpenOffice respectively. Nice try though.

          That said, Google's results are still generally better.

        • I don't know what you're trying to prove. For me, the first link for "getfirfox" is the MoFo/Google getfirefox.net. The first link for "Linux is linux.com. And the first link for "Open Office" is OpenOffice.org.

          Now if you want to argue that the "sponsored links" are crap, I can't help but agree. But Google has the exact same problem.

        • Now I remember why I had never bothered to use their search. I expected tricks like that and there they are. It's funny to note they didn't buy/reserve that misleading "Linux" ad that points to Windows for the two word search term ubuntu linux but they did for the one word ubuntu which has more meanings.

          It's also funny to note how much more the interface looks like google's than anything MS has done before. It's so obvious they are admitting they don't know how to innovate anything themselves.

      • Re: (Score:3, Insightful)

        by Anonymous Coward

        i wish they would work on weeding out the crap.

        There are a *lot* of people at Google working on that problem. Please understand that it is really difficult to keep up with new attacks when your site is #1, because many people out there are aiming directly for it. No matter how many work on ranking and relevance inside the company, there will always be 10x-100x that number of people outside who are working on the shady side of SEO, spamming, etc. It's a never-ending battle, much like spam email. We're trying.

        anything you google now is infested with cheesy search sites that list other websites and try plaster you with ads. they contribute zero to the web.

        We're working on that both from the search

    • They rub the servers with cheetah blood. Anyway, the real magic is how is their index that big when there's not that many websites? I thought I just saw an estimate like a half a year ago saying that there's about 4 billion or 10 billion or whatever domain names registered and they estimate xxx number of total web pages and it wasn't even close to 1 trillion. Did all those stupid web 2.0 pages screw with it that much?
      • I imagine that certain sites, such as sites the size of Slashdot (in terms of dynamically generated pages), make a difference. After all, the index talks in pages, not domains. I bet there's also a lot of junk and redundancy in there, but still, it's quite an achievement to be able to deal with that much data.

        • Re:Amazing (Score:4, Funny)

          by cammoblammo (774120) <cammoblammo.gmail@com> on Saturday July 26, 2008 @02:38AM (#24346147)

          I imagine that certain sites, such as sites the size of Slashdot (in terms of dynamically generated pages), make a difference. After all, the index talks in pages, not domains. I bet there's also a lot of junk and redundancy in there, but still, it's quite an achievement to be able to deal with that much data.

          Surely you're not saying that Slashdot's full of junk and redundancy and redundancy?

    • by KillerCow (213458)

      As someone who is partially engineering/analytically minded (but not a great programmer) it amazes me how Google has manged to index so much data, yet at the same time, serve up results in a fraction of a second to so many people.

      See "map reduce"

      • by tuomoks (246421)

        Thanks - I was going to reply but you got there first. Map reduce and big table, etc - old technologies found again. A must when you have to match arbitrary queries - SQL (actually relational databases but in many minds same?) just doesn't do it, something what was know, forgotten and slowly acknowledged again.

  • by Anonymous Coward on Saturday July 26, 2008 @12:17AM (#24345559)

    Seriously, since the web is something like 42% porn. (Yes, that is the ultimate answer.) So that's on average, 60-70 pages of each person in the world naked.

    • by sweet_petunias_full_ (1091547) on Saturday July 26, 2008 @01:46AM (#24345959)

      "the web is something like 42% porn"

      That probably stopped being the case after namespace speculators started buying up expired domains in large numbers just to put up a mildly useless index on *each* and *every* site to collect ad revenue or marketing statistics off of unwary visitors. I would also include typosquatters in that category, and maybe someone else can name a few other examples of utter namespace hogging uselessness.

      Whatever it is, you can rest assured that it's mostly repetitive trash... no need to stand in awe of it.

  • by jollyreaper (513215) on Saturday July 26, 2008 @12:17AM (#24345561)

    How many of those are automatically generated rank-spoofers, 80%?

    My favorite spoof pages were the ones that randomly substituted search terms into porno stories.

    "Yes!" she screamed as he thrust his SAMSUNG CD PLAYER deep into her. "I want you balls-deep in my CHEAP HARD DRIVES!" The smell of DISCOUNT SOFTWARE filled the room.

  • by Anonymous Coward

    Trillion can mean 1E+12 or 1E+18 depending on which country you are in.

    • by kclittle (625128) on Saturday July 26, 2008 @12:47AM (#24345735)
      Google is headquartered in Mountain View, CA -- I know, 'cause I googled it. Now, California is rather inclined to think of itself as it own country (some would say, universe), but it is indeed part of the United States of America (again, I checked with Google). And in the US, "trillion" == 1E12 (again, Google).
  • by consonant (896763) <shrikant,n&gmail,com> on Saturday July 26, 2008 @12:23AM (#24345613) Homepage
    ..knowing that the vast amounts of porn just keep getting vaster. And more searchable. Amen. *sheds a tear or two*
  • Some numbers (Score:5, Interesting)

    by Reality Master 101 (179095) <RealityMaster101@@@gmail...com> on Saturday July 26, 2008 @12:30AM (#24345661) Homepage Journal

    Counts of words:

    the: 18.3 billion pages
    a: 23.9B
    0: 12.7B
    1: 25.4B
    in: 17.1B
    I: 10.2B

    I know these numbers aren't exact, but you'd think one of them would be over 100B if Google is really indexing a trillion pages. What's on them? Anyone find any keywords that produce more?

    • by Shaitan Apistos (1104613) on Saturday July 26, 2008 @12:44AM (#24345721)

      My hobby:

      Getting the fewest possible google results above 0 with a quoted string.

      "interspecies gangbang": 6
      "hot topic meets disney world": 2
      "died in a blogging accident": 15,300
      "can boys make babies": 4
      "why does it hurt when I read": 1

      • Hmm. This probably means something statistically, but I'm not sure what... I started adding digits to a number until I hit one result. I had to get to nine digits:

        123512553
        215323703
        684354537

        I also found a few with 0 to 3 results. Interestingly, I couldn't find any eight digit numbers that scored zero hits.

        • by txoof (553270)

          I couldn't find any eight digit numbers that scored zero hits.

          That's really interesting. Perhaps 8 digit numbers are common for serial numbers and dates. Today's date is 20080723. I can't even begin to think of all the logs and file names that I've generated that contain a similar string.

          Anyone else have any ideas why 8 digit numbers are so common?

          • Re: (Score:3, Interesting)

            by SnowZero (92219)

            You can list all of them with less than a gigabyte: 10^8 * (8+1) ~= 858 MB
            The web is pretty big, so all of them are bound to happen *somewhere*.

            Plus, I just registered all8digits.net

      • The only way to save your accomplishments is to not post them on the internet
        • by arotenbe (1203922)

          Indeed. Google already has two hits for the phrase "why does it hurt when I read": the actual (and rather disturbing, but amusing) page the quote was found on, and this page. Scary, isn't it?

      • Re: (Score:3, Insightful)

        by miraboo (1164359)

        My hobby:

        Getting the fewest possible google results above 0 with a quoted string.

        "interspecies gangbang": 6
        "hot topic meets disney world": 2
        "died in a blogging accident": 15,300
        "can boys make babies": 4
        "why does it hurt when I read": 1

        My Hobby

        Attributing my sources: http://xkcd.com/369/ [xkcd.com]

      • It's harder to do now. If you can find two words unquoted that result in one result it's a googlewhack [googlewhack.com] like this one was [google.com] before Google found "eltiguan parainaugurarme" on this page and made it the second result.

        BTW, my first google search was "war" and it returned something equivalent to "Your search term is too common to return a meanignful result. Narrow your search." Today it returns about 974 million results. It was long ago...

      • Re: (Score:3, Informative)

        by Ihmhi (1206036)

        You mean Googlewhacking [wikipedia.org], except not nearly as hard?

      • by spacefrog (313816)

        Much like Fight Club, the number one rule of your particular hobby is not to talk about your results hence you taint them.

        Kind of like counterfeiting currency or collecting teenybopper porn, it sucks to have a cool hobby you can't brag about.

        • by kv9 (697238)

          Kind of like counterfeiting currency or collecting teenybopper porn, it sucks to have a cool hobby you can't brag about.

          I brag about that all the time.

    • Or maybe a lot of those pages are in non-Latin script. According to the Chinese government there are more Chinese online than Americans, and they don't use the latin alphabet....
      • I thought maybe different languages might explain things, but I'd say English is by far the most popular languages. I might even buy English is not the majority of pages anymore (though, I'm skeptical), but only 1 in 25 pages is English?
    • by WK2 (1072560)

      I tried the following: the OR a OR 0 OR 1 OR in OR i

      It says "Results 1 - 10 of about 0". Must be a wrap-around bug.

  • by bogaboga (793279) on Saturday July 26, 2008 @12:31AM (#24345663)
    This might be off-topic but I wonder what's going on with Sergey Brin and Larry Page's [PhD] education? Just wondering...did they give up?
    • If I turned billionaire before I finished schooling, I don't think I would finish unless I got bored enough to do so. I don't think it's necessary for them to finish either, because they can hire plenty of PhDs with the knowledge that they would have gotten if they finished.

      Running a business doesn't require that you know everything needed to run the business, only that you know how to get people with the skills that you need.

  • No, it didn't. (Score:5, Informative)

    by aiken_d (127097) <brooks&tangentry,com> on Saturday July 26, 2008 @01:06AM (#24345809) Homepage

    They have identified that there are 1T pages out there, somewhere. They have indexed 40 billion pages. Read the entire Google post. It says it right there.

    Bad on Google for the misleading post. Bad on the submitter for not reading the misleading post. Bad on Slashdot for further descending into mindless repetition of mindless submissions of mindless PR announcements.


    • They have indexed 40 billion pages. Read the entire Google post. It says it right there.

      I didn't see that information in either post, only that their index is somewhat less than the full 1T unique URLs. The full size of the Google index is probably confidential, proprietary info. Their competitors could use it as a benchmark for assessing how their service compares, so I'm not surprised that it's not there--or if it was there that it was removed.
  • by Animats (122034) on Saturday July 26, 2008 @01:25AM (#24345867) Homepage

    But how many of those trillion pages have unique, useful content? E-mail is over 95% spam, and the web is getting there.

    There were about 153 million registered domains at the beginning of the year. The ones from the spam-friendly registrars [knujon.com] are mostly junk. Tim Bernars-Lee said in 2006 that web junk was becoming a major problem, and it's become worse since then.

    If you throw out all the anonymous but commercial domains (we call them "bottom-feeders"), as we do with SiteTruth [sitetruth.com], the Web looks a lot better. Search engines are getting stricter about this. You don't see that many "landing pages" in Google any more. Bad news [fool.com] for companies like Marchex [yahoo.com], the publicly traded web spammer that cranks out all those junk "What you need, when you need it" sites.

    "The mass trials are going well. There will be fewer Russians, but better ones." - Greta Garbo in Ninotchka.

  • A trillion URLs, and still no sign of clownpenis.fart in the index anywhere!

    At this rate it really will be the last one to go.

    • by repvik (96666)

      Huh? I just *had* to google that one...
      Results 1 - 10 of about 8,580 for clownpenis.fart

  • Specifically which page was the trillionth?

  • by blind biker (1066130) on Saturday July 26, 2008 @03:57AM (#24346391) Journal

    I think google.com's search engine achieved its peak usefuleness about 5 years ago. Now, for the most part when I google for a certain electronic component I get some crappy webstore front (and by crappy I mean I can't actually order the component but must "contact by phone" first) or if I search for an electronic device, be it pro or just home electronics, I get those "Read reviews and compare prices"-sites. Which I hate with a passion. WTF google, you have the world's most talented programmers, can't you weed out this crap from your search? At least so it doesn't come up as top hits?

    • It's almost like google stopped bothering about returning relevant commercial sites in the main search, and farmed it off to some subsidy. I mean, talk about saving money, that's a very froogle thing to do.

      Please then allow me to bludgeon you with the point, which is google's most excellent product search at www.froogle.com . If it's buyable on the internet, it's there.

      • Re: (Score:3, Interesting)

        by blind biker (1066130)

        Thanks, but what I was trying to say (and I'll admit to bad wording), is that not only does google.com search return webstore fronts when I am actually looking for technical information about electronic components (this is the point I did not get across well - I am not looking for shops, but for info), but it returns the worst kind of webshops. The kind that isn't really a webshop at all, as in, you can't actually buy anything from them using the web.

        As for froogle: I just tried searching for "NAD 701" (wit

        • Ah yes, technical information about something that gives you 5 pages of crap, even with "technical manual" specified in the search, is a problem. quite right. Of course, if the manufacturer hasn't released it in the first place, it might just not be on the internet.

  • I mean, really. 90% of it is junk.

     

  • by Coolhand2120 (1001761) on Saturday July 26, 2008 @05:52AM (#24346759)
    There are so many dynamic pages on the net now that one web site, like slashdot as an earlier poster commented, can contain literally millions of pages. People use programs like modrewrite [apache.org], isapirewrite [isapirewrite.com] and linkfreeze [helicontech.com] to manipulate spiders into crawling pages that are near identical. For more than one customer I've made meta, title and content randomization, serialization and or URL rewriting schemes to make damn sure spiders index every possible dynamic page, and it works. I have a single dynamic page that must have been indexed hundreds, maybe thousands of times with slightly different content, and they are all in the index.

    Google tries to detect a dynamic page by looking for ampersands and equal signs, as well as looking at the content of the page, it is really quite easy to fool.

    e.g.: http://somesite.com/itemlist.php?listmode=1&category=beds&orderby=7 [somesite.com]
    when 'rewritten' shows up as
    http://somesite.com/items/1/beds/7.html

    So 1 billion web pages could be, and I know a few thousand pages like this, just a few hundred thousand dynamic pages. Not that the pages don't have relevant information, some of the stuff can be redundant though. For instance, when the spider crawls across "Records per page = 10" > "Records per page = 20" > "Records per page = 30" etc.. or when lazy programmers don't use cookies and databases to store information but try and concatenate the URL with the user's selections. Thank god for that GET limit [boutell.com]. People need to use POST!

    If someone knows how to stop this message board from creating links out of false URLs please, let me know.
  • "Why is McDonald's still counting? How insecure is this company? Forty million eighty jillion killion trillion....is anyone really impressed anymore? Oh eighty-nine billion sold! All right I'll have one."

    -- Jerry Seinfeld

  • google finally has a use for a bigint auto-increment primary key now!!!!

The economy depends about as much on economists as the weather does on weather forecasters. -- Jean-Paul Kauffmann

Working...