When it comes to search results, is more always better? Considering that most of us don’t even look beyond the first page, how many results do you actually need? Does it matter that most of that big impressive results number isn’t actually accessible?
There’s an active debate going on over the number of returned vs. viewable results on the major search engines and whether or not the engines are presenting the whole picture.
Sparked by this piece by Tristan Louis which compares the search indexes of Google, Yahoo, and Technorati, many in the blogosphere are wondering why a query’s reported results are often so much larger (up to 1,000 times) than its viewable results. Is this deliberate deception on the part of the search engines? Pure laziness from the engineers?
Dave Sifry, CEO of Technorati, is asking all of these questions and then some in this post titled On Search Engine results comparisons: Where’s the remaining 99.8% of the results? In it he conducts a sample search and spot checks the link results for Google, Yahoo, and Technorati. Conveniently, he finds that his own company is the only one providing access to the full range of results – Google and Yahoo max out around 0.2%.
While we share some of Dave’s concerns over the need for more accurate labeling, we agree with Danny Sullivan that this is nothing new. None of the major engines allow you to search past the first 1,000 results as a matter of practice. It takes too much computing power, power most of us would rather have put into relevancy than comprehensiveness. After all, it’s not like the results really stop at 1,000 – one page leads to another, that page leads to another, and so on.
Back in 2000, Excite made this statementon their site, one which other engines would do well to follow:
As a highly trafficked site, Excite has a responsibility to provide the best possible service to our customer base. Our Excite Search engine is designed to float the most relevant data from our index to the forefront of the results list returned for each query submitted. We have found that nearly 100% of users never have a need to drill down beyond the 1,000th result for a given query. For these reasons, we no longer provide more than 1,000 results per query submitted. We hope that 1,000 results more than meet your needs.
For more, check out the Search Engine Size Test on Search Engine Watch. Although it’s old, the methodology holds up.
