Was Bing it On Biased?

Between December last year and June this year, Bing ran a survey that led it to conclude that most people prefer their search engine to Google when they do not know which is which. Now a new study has raised questions about the validity of that survey, and the Bing team has found itself on the defensive.

The second study was undertaken by blogger Ian Ayers and four law students from Yale. Like Bing, it involved presenting participants with the two search engines in plain form, so that they looked the same, and asking them to run searches on both and say which they liked best. It did this using search terms from the same list Bing had used, but it also asked users to make their own searches or to run the most popular searches at the time. When they were not using search terms from Bing’s list, the participants did not rate it as highly; in fact, the results were reversed.

Did the Bing study set out to skew the results of the original survey? Bing has denied this and notes that the study was conducted on its behalf by an independent company, Answer Research. The terms used were in fact drawn from Google’s Zeitgeist list, the idea being that if they were popular then participants would be more likely to know about them and would be able to assess the search results they got more effectively.

What does this mean for search engine optimization specialists trying to weigh up whether Bing or Google is the most important long-term bet? From this point of view, it does not really matter what Bing’s intentions were – if the survey results were biased, it is difficult to base any useful conclusions on them.

Other data, however, does show that Bing is gradually chipping away at Google’s massive market dominance, and all the more so if its partners – such as Yahoo – are taken into consideration. The Yale students’ survey suggests a 35% to 39% preference rate for Bing on a blind test, which is above its current market share and suggests that there is still quite a bit of room for it to grow before it has to start making any major technical changes to win over new users. Google is likely to remain number one for some years, but its position can no longer be considered unassailable.