DuckDuckGo Surpasses 100 Million Daily Search Queries For the First Time (zdnet.com) 56
Privacy-focused search engine DuckDuckGo reached a major milestone in its 12-year-old history last week when it recorded on Monday its first-ever day with more than 100 million user search queries. From a report: The achievement comes after a period of sustained growth the company has been seeing for the past two years, and especially since August 2020, when the search engine began seeing more than 2 billion search queries a month on a regular basis. The numbers are small in comparison to Google's 5 billion daily search queries but it's a positive sign that users are looking for alternatives. DuckDuckGo's popularity comes after the search engine has expanded beyond its own site and now currently offers mobile apps for Android and iOS, but also a dedicated Chrome extension. More than 4 million users installed these apps and extension, the company said in a tweet in September 2020.
Better results (Score:5, Interesting)
Re: (Score:2)
Image search is also just about identical, and I'm kind of fed up with Bing/DDG returning 80% YouTube video links in the image searches. Looks like all the DDG/Bing searches are filtered through Google SafeSearch, so whatever Google deems verboten affects all these engines. Can anyone confirm this?
Re:Better results (Score:4, Informative)
I get identical results between DDG and Bing. They're the same engine but different brands.
If you want the same results as Google but with extra privacy, you can use Startpage.com [startpage.com].
Re:Better results (Score:5, Interesting)
On woke topics, google steers me in the woke direction, while DuckDuckGo gives results offering a more balanced viewpoint.
I noticed the same poisoning of the Google search terms on many political and COVID topics. This is considering that I take effort to wipe all tracking, so you can't even attribute skewed result to the previous search history.
Re: (Score:1)
Re: (Score:1)
"I noticed the same poisoning of the Google search terms on many political and COVID topics."
That's only because you're a right-wing nutjob.
I didn't get the memo that Google now is only for left-wing nutjobs. My bad.
Re: (Score:3, Funny)
Dear google, please return only search results that tell me what I want to hear.
Re: (Score:2, Troll)
Dear google, please return only search results that tell me what I want to hear.
How would you explain DuckDuckGo presenting additional search hits that Google does not have? It is very clear that Google is suppressing search results.
Re: (Score:2)
Dear google, please return only search results that tell me what I want to hear.
How would you explain DuckDuckGo presenting additional search hits that Google does not have? It is very clear that Google is suppressing search results.
Easy, to me it's been very clear for about a year now that DDG has been artificially promoting far-right sites. I've often noticed with DDG that my initial search using their safe mode in a fresh browser will return different (and more numerous far-right) hits than subsequent searches will if I move to a less restrictive mode and back again to the safe one. After which time, the safe mode hits seem more logically distributed.
Re: (Score:1)
Re: (Score:1)
The alternative explanation is that you are so deep into far-left echo chamber that you lost your compass and can't tell apart unfiltered search and far-right propaganda - because it looks all the same to you.
Your psychic skills in divining my politics are "deplorable".
The most likely scenario is that both DDG and Google tailor their results and derive their initial baselines upon models that result in a particular bias; offering completely alternate realities to whomever they think the user is, while they pretend to offer a single unified "reality" to everyone. Facebook does the same thing. Feel free to pretend this is about my politics and limit yourself to one of those search engines though; I'm sure your ins
Re: (Score:2)
The most likely scenario is that both DDG and Google tailor their results and derive their initial baselines upon models that result in a particular bias;
Do you have any evidence to support your claims about DDG? You are making extraordinary claims - that DDG tailors default searches to far right audience. What makes you say that?
Re: (Score:2)
Re: Better results (Score:3)
Did you know that Google bases its result sorting on the area you're in, time of day, day of the week, etc... And of course on what you did in the past, including e-mails.
And it has a really hard time telling pro from contra or recognizing sarcasm. (Meaning it completely fails.)
So can I guess you're living in a very woke area, maybe share your wifi or computers with woke people, or lookd up many sites regarding that topic but with a different polarity?
I found disabling ALL the "personalization" gets rid of
Re: (Score:3)
Re:Better results (Score:5, Informative)
I don't use DDG for the better privacy, but for the better results.
Relatedly, DDG's biggest advantage to me is its support for bangs [duckduckgo.com]. if I want to check what the critical consensus is for The Mandalorian, I can type "!rt Mandalorian" into my search bar on any device to immediately jump to the Rotten Tomatoes results for the search term. I use it all the time when I know I want to go straight to StackOverflow (!so), check for used products (!ebay), look up who that actor was in that movie (!imdb), read up on a subject on Wikipedia (!w), pull up something on Amazon (!a), and the list goes on and on. It makes the bulk of the searches I do so much easier.
DDG even provides a !g for those times when you think you might have more luck with Google (as well as !gi for Google Images, !gm for Google Maps, etc. if you aren't finding the results you thought you would).
And that's before we get to the privacy benefits, such as permitting POST instead of GET for submitting search terms, removing search terms from link referrals (i.e. keeping sites from knowing what you searched for), having default settings that protect users against the privacy implications of embedded content from places like YouTube (though you can choose to enable embedded content if you trust those sites), and so on.
Re: (Score:3)
What is the advantage of doing that over just using the browser's built in feature? Both Firefox and Chrome support it natively. Seems like your way of doing it pings DDG every time you use it.
Re: (Score:3)
If you're willing to put in the time, your approach is arguably better in some ways, sure, but it takes manual effort to add each site, whereas bangs provide tens of thousands with zero effort. Instead of requiring setup that I either need to do in each browser or sync to each browser, bangs work whether I'm using Brave at my computer, Safari on my phone, Firefox on my wife's laptop, or Chrome when troubleshooting my parents' computers.
But if pinging DDG is the concern, you can find extensions that will do
how is the chrome extension private? (Score:2)
Doesn't chrome ship everything you put into the omni bar to google? and all your bookmarks?
I'm pretty sure it also uses the google DNS no matter what your settings are. I once misconfigured my system wide DNS but chrome kept working.
Re: (Score:3, Insightful)
If you are relying on a search engine to give you a balanced overview of a controversial topic then you are doing it wrong.
Do you imagine that someone at DDG has carefully selected search results for your terms to make sure you aren't mislead? Or that their algo is smart enough to understand the subtleties of what it is reading and give you a balanced selection?
At best every search engine's result just reflects the biases of the people who decide what "good results" means. At worst they give you the most he
Re: (Score:2)
Same here (Score:1)
Re: (Score:1)
Just type "men can" and watch the autocomplete magic happen... For those who are to lazy to try out out themselves, the results are:
Man can have periods
Man can lactate
Men can get pregnant
Re: (Score:1)
Re: (Score:2)
DDG (preferred for privacy reasons) gives me bad results compared to Google though for me. :(
Is there a way to measure index completeness? (Score:5, Interesting)
E.g. how much does their index contain, compared to Google?
(To be fair: Compared to what Google would see, if it didn't spy on literally all e-mails and everything else that should be private.)
I mean with actual numbers. Not "It feels about as complete / less complete.".
Asked in another way: What stops DDG from crawling just as much of the web as Google? (Again, given the above caveat.)
I figure the only difference is goingnm to be people actively telling Google that their site exists, and actively checking it's in the index... but not DDG.
And I feel that it was an error in the design of the web and DNS, to not have a central (well, shared) index, and making it mandatory to offer a complete crawling index of what should be publicly available, if you want to register a domain. (You could still choose to offer an empty index if nothing should be found.)
Something legislation could easily fix, killing any search engine's index dominance, and letting them focus on the quality of the actual search. (Matching your input to index entries. And presentation.)
Re: (Score:2)
Completeness is only one metric, another important one is how up to date the index is. Results appear on Google within minutes or even seconds of going up, very handy for following current events or not getting bad links to sites that have removed the content you were looking for.
Having sites offer their own crawling index is a bad idea. They will just lie to get clicks. The law is irrelevant, they will locate themselves out of your jurisdiction, if you can even find out who the owners are.
Re: Is there a way to measure index completeness? (Score:2)
They already lie. Look at link farms. And links and keywords pointlessly added.
That does not make it a bad idea.
Anything done with an official index can alreqdy be done.
And I did imply that the crawler would ust that index as merely starting points. I was thinking of robots.txt.
Re: (Score:1)
That would be a nightmare. If the index were only required for registration, it would soon become obsolete. If the index had to be kept up to date it would be a huge make-work project. How would the requirement be enforced?
Re: Is there a way to measure index completeness? (Score:2)
/Obviously/ you'd make up-to-dateness a requirement.
Since sites also *want* their content to be found, they would be interested it keeping it up to date. (Again, think robots.txt, not a full list. Only starting points.)
Any site not even able to do that, would deserve not being in the search engine's index.
Re: (Score:3)
Granted, it has not worked out that way, but the ideal is still sound.
Re: Is there a way to measure index completeness? (Score:2)
I am not.
Straw man much?
I say everyone should have a good robots.txt. Nothing more.
DNS is already centralized, in case you hadn't noticed. By your government. I oppose that. (I'd also oppose it, if it was mine, or China's.)
TLS is centralized by browsers too. (I prefer webs of trust.)
Re:Is there a way to measure index completeness? (Score:4, Interesting)
Is there a way to measure index completeness?
Kinda, but not really.
I did my graduate research in this field back in the mid-to-late 2000s (my research group did the then-largest web crawl in academia, with an index roughly 20% the size of Google's at the time). I'm sure things have changed in the last decade, but my recollection is that we were putting more focus on metrics like pages crawled, sites crawled, and a variety of metrics for the freshness of an index, rather than anything resembling "completeness". We did measure how many pages we crawled of the links we had seen, but the idea of completeness is somewhat meaningless in the face of an infinite Internet. For example, the number of articles on Wikipedia may be finite, but there are an infinite number of pages you could potentially crawl because the search results page repeats your search terms back to you, of which there are infinite permutations. Even something as simple as a phpBB forum that displays the current time is, with nearly every refresh, a different page that would need to be indexed if we wanted 100% completeness. Same for a calendar app that can be infinitely navigated into the future.
But we all know that we don't care about those things. It's just a matter of where you draw the line between useful and useless.
Really, crawling "everything" is not particularly complicated. You need time and/or a fast network connection, but even relatively inexpensive hardware can get the job done (we ran our crawl from a commodity server with a RAID, but we were granted essentially unfettered access for over a month to the network connection serving one of the largest universities by student population in the US). The harder part, which is where my research was focused back then, was in knowing what and when to re-crawl. The long tail of the Internet is something you could for the most part crawl just once and then rarely if ever need to crawlagain, but the leading edge is something that needs to be re-crawled constantly to keep your index fresh. With a limited number of resources (even for Google), it's massively important to understand where to focus your efforts, otherwise you'd end up re-crawling Joe's Blog that hasn't been updated since '01 as frequently as you'd re-crawl Twitter or the front page of CNN.
But once you reach a certain point (which all of the major players have reached, I'd wager), completeness stops mattering nearly as much. Instead, how you use that data is what matters. For instance, even if a search index had 100% completeness and was magically updated anytime someone did anything online, it'd be useless to people if users had to know the hash of the content they were interested in. It needs to be useful to people, and I suspect that's where Bing (which supplies results to DDG) is falling a bit short still.
The science around search gets out of my wheelhouse in a hurry, and I'm sure the state of the art has advanced significantly since I was up on the subject, but there are all sorts of techniques for making pages searchable in useful ways. After all, you can't scale naive text search for all content on the Internet to a size like Google's without each search taking an inordinate amount of time. It's simply too much to search through. As such, you may use quick-but-rough techniques to knock the list of potential results down by several orders of magnitude, then use more expensive techniques to refine the results with whatever is most applicable to the user's search term(s). You may combine multiple, quick methods and then try to determine the relevancy of individual results returned from each method in creating a final set of results for the user. If the user searches for quoted text to which they expect an exact match, the techniques you'll employ and the indices you'll use are going to be very different than if you're trying to give them meaningful results to a search for "jaguar claw", since you somehow need to understand from the context that they're referring
Re: Is there a way to measure index completeness? (Score:2)
There are not infinite search results. You are implying a crawler would also present an infinite number of input parameters. In reality, it doesn't, and there is only one single actiul search page. I That is precisely why such parameters go in the HTTP POST, not GET. Even if some so-called developers never heard of that. A crawler would not just search for every character combination possible. It would click search IFF it was a GET method form, or ignore the button if it was a POST one.
Re: (Score:2)
There are not infinite search results. You are implying a crawler would also present an infinite number of input parameters. In reality, it doesn't, and there is only one single actiul search page.
I didn't say there were infinite search results. I said there were infinite pages on the Internet, which may seem like a pedantic distinction, but actually isn't. Any page or content presented in the browser that has different content than any other page and a different URL than any other page is a different page, no matter how trivial those differences are. You and I may know that the site is using the same view in the backend to display different results or search terms, and we may even understand that we
Re: Is there a way to measure index completeness? (Score:2)
You completely failed to mention where you'd get the crawling starting points in the first place.
I don't think you can even walk all of DNS right now. Did you get access to all registrar databases, to be able to list ALL domains under e.g. .com, and under all other domains too... even .weirdtldwithanunreachableregistrar?
Re: (Score:2)
You completely failed to mention where you'd get the crawling starting points in the first place.
You asked and I answered a question about completeness. Starting points aren't particularly relevant to that topic, nor do they particularly matter. A site is either connected to all of the other sites, or it isn't. If it is, a crawler should eventually reach it. If it isn't, a crawler generally won't reach it.
I don't think you can even walk all of DNS right now. Did you get access to all registrar databases, to be able to list ALL domains under e.g. .com, and under all other domains too... even .weirdtldwithanunreachableregistrar?
Nope. Like everyone else I'm aware of, we just crawled the publicly available pages that are linked from other publicly available pages. I mean, I suppose you could crawl whois by entering random term
No captchas (Score:5, Interesting)
No Doodles! (Score:3)
DDG sadly getting worse (Score:5, Insightful)
I have DDG as default search engine on all my devices and it will never change but I have to say that last months I see me double checking google more and more often. DDG works as a front end for youtube, wikipedia and most of the times for technical searches but for many topics it's absolute garbage. For example, the other day I was looking for Covid emergency number in my town. I go to DDG:
https://duckduckgo.com/?t=ffab... [duckduckgo.com]
first link is from Mexico, there are some pages about restrictions and no phone number for my town. Now google:
https://www.google.com/?q=nume... [google.com]
The first result is exactly what I'm looking for, thank you. And that while being logged out and with cookies cleared after every search.
I value privacy and I'm willing to search twice from time to time so I'm not going back to google as default search but it's really hard to convince other people to use it which results this bad.
Re: (Score:2, Funny)
For example, the other day I was looking for Covid emergency number in my town.
You sir, are a Brainiac.
Re: (Score:2)
Yeah, google is tracking me to provide info for a town whose name I included in the search. How else could they figure out which town is it? Other than analysing the query of course.
Not worse, not better - just different (Score:2)
For most searches, DDG is fine. On any topic that could attract paid search placement, Google is terrible - it's all commercial websites, where DDG will get you informative articles. For localization (phone number of the restaurant around the corner), Google is better.
I start with DDG, and use Google only if necessary.
Re: (Score:2)
I have DDG as default search engine on all my devices and it will never change but I have to say that last months I see me double checking google more and more often. DDG works as a front end for youtube, wikipedia and most of the times for technical searches but for many topics it's absolute garbage. For example, the other day I was looking for Covid emergency number in my town. I go to DDG: https://duckduckgo.com/?t=ffab... [duckduckgo.com] first link is from Mexico, there are some pages about restrictions and no phone number for my town. Now google: https://www.google.com/?q=nume... [google.com] The first result is exactly what I'm looking for, thank you. And that while being logged out and with cookies cleared after every search. I value privacy and I'm willing to search twice from time to time so I'm not going back to google as default search but it's really hard to convince other people to use it which results this bad.
Bingo. I don't use DDG except for simple things (wikipedia, youtube, imdb front end.) Anything more complicated or "semantic", then it's google (or bing.)
Re: (Score:2)
Same problem here - the biggest issue I have is location sensitive results. Google hands down wins here by miles. I suspect part of it is the usual American bias (I'm in Australia) where more popular search results are overriding my local ones, but I also attribute a big part of it to the fact that (in theory) DDG doesn't know where I am or where I live. Unlike Google which knows my home and work addresses (thanks Google Maps) and my current location and travel history (thanks Location Services), and of cou
Re: (Score:2)
IP based location is very simple and everyone does it. You don't need Location Services or account in Google Maps for it. For me the fact that DDG doesn't use it at all is crazy, it ruins way too many searches.
Re: (Score:2)
Creepy vs. useful (Score:2)
That "free lunch" with Google can be really tasty. The more info they have on you, the more the results chime with what you are searching for. ... WTF?
The creepy side of that is when you start typing something and auto-fill completes it for you
I mean, it's crazy useful, but you just know it is driven by what Google knows about YOU - that's the price you pay for accurate results.
But, on the flipside, doesn't that mean it just brings up results that match how you are profiled and you miss a bunch of hits in t
Firefox (Score:5, Interesting)
I’ve noticed fresh installs of Firefox are defaulting DDG. That will certainly help adoption.
Don't Be Evil (Score:1)