Disclaimer: I work for Bing as a senior research software developer, but I do not speak for Bing; I am writing this on my own time and behalf.
Google has accused Bing of cheating, of copying Google’s results (Danny Sullivan’s article which broke the news; Google’s official blog post with the accusation). Basically, Google set up a honeypot of synthetic (that is, unique, non-word) search keys which their results led to specifically selected (i.e., non-algorithmically), but real, websites. Later, these search queries, when issued on Bing, led to these selected results. Bing’s Harry Shum (the Microsoft VP in charge of Bing) had this to say in an official response to Danny Sullivan’s article:
We use over 1,000 different signals and features in our ranking algorithm. A small piece of that is clickstream data we get from some of our customers, who opt-in to sharing anonymous data as they navigate the web in order to help us improve the experience for all users.
When a search engine, like Bing’s or Google’s says the results are algorithmically chosen, this primarily means that machine learning algorithms are used for selecting a collection of results to potentially display, and then other machine learning algorithms are used for ranking the results for display (which goes first, second, etc.) Of course, there are a lot of other things that go on: algorithms are used for spelling correction, query alterations (for example, noticing that “SF” is a common way of saying “San Francisco”), query classification and answer type selection (for example, should “365/24/7” return a direct answer of 2.17261905?), and so on. But what Google is accusing Bing of doing is “cheating” by copying their answers directly from Google, which is to say, that the usual selection and ranking steps are being bypassed in favor of direct copying.
Both Google and Bing use machine learning models that use many features for selection and ranking. This is what was meant when Shum said Bing uses “over 1,000 different signals and features.” A subset of those features are “clickstream data we get from some of our customers, who opt-in to sharing anonymous data.”
The clickstream (requisite Wikipedia article) is the record of searches, displayed results, and selected results collected by “our customers,” as Shum said. Clickstream analysis (pioneered by Google, I say, admiringly) is an extremely powerful source of data for search result selection and ranking. It tells you, implicitly, what people think of the results presented to them. Given a ranked selection of A,B,C, they click on C and then B, but leave A alone. Given enough of these data, the machine learning models can learn to present C and B (in that order) and downrank A (if A is presented at all). And there is a lot of clickstream data, both in the direct logs to the search providers’ services, as well as the “opt-in” data mentioned by Shum. Obviously, Bing can’t inspect Google’s clickstream logs, but when customers allow it, Bing can use their searches made to Google to approximate this. I don’t know the details of what is collected (nor could I tell you, I suppose, if I did), but these are the data Shum is referring to.
Ok, so imagine you have a complicated function that takes “over 1,000 signals and features” as its input, to use for ranking or selection. But in some very specific cases, you only have only a few signals coming in; the rest are unknown. Typically, and algorithmically, the function will do what it can with the information it has. In the present case, if the only thing it knows is that several clicks on a particular website occurred when a set of users entered in queries (and never any other websites), the function will likely return that website; a completely reasonable thing to do. It should be noted that Google’s honeypot trap resulted in only about 10% of the results, which is consistent with the paucity of the data, the strength of the signals, and the time lags involved.
To my mind, this is not “copying,” but the natural result of algorithmic search. Google’s results are a kind of gold standard; that the machine learned models learn to do what Google does is not unexpected. One way for Bing to “beat Google” at search is to be as good as Google in algorithmic search; which implies being as good as Google first. I don’t think you’d get an argument from anyone that Microsoft Live—Bing’s predecessor—was not as good as Google in core search. But I think a lot of people are beginning to realize that Bing’s core results really are about as good as Google’s core results now. And this is a result, not of copying, but of all the engineering and hard work of identifying the “over 1000 signals and features” that go into ranking and selection.
Interestingly, Google’s accusation came out the evening before Matt Cutts (who helps Google maintain search quality and lead anti-spam efforts) and Harry Shum were to appear jointly at “Future of Search” roundtable discussion (along with the CEO of Blekko, a new search engine). Cutts accused Bing directly here, and Shum said more or less what he said in the blog post. But he said something else interesting. He said that Google’s honeypot was a new kind of query spam (and that he was glad that Google had identified it). Maybe this was just trying to get a dig in at Google’s most public anti-spam advocate. But there really is some truth in this. Having identified this kind of query trap, I suspect engineers at Bing will look for ways of turning this into an anti-feature, and these sorts of results will be learned away.
You could tell that Cutts, as mild mannered a person as you’re likely to meet, was really upset at the thought that Bing is unfairly copying Google’s results—in fact, he said as much at the round table. I don’t quite get this, and I’d like to understand it better. He knows better than most how these things work. When I first started working for Microsoft (after the acquisition of Powerset, a startup company I was part of), I was suspicious. What I found is a large group of dedicated search engineers who want to build the best possible search engine, measured and driven by data—a method modeled for us by Google. I guess we’d be upset if we thought a competitor were cheating by using our results. But, from my standpoint, Shum’s public statement about what Bing is doing fairly describes what is happening: Bing uses clickstream data in its models, which sometimes leads to similar selections and rankings as Google’s selections and rankings. Bing isn’t cheating or copying; it’s using data from customers (who have opted in) to improve its results.
Will.Whim 
Will, an interesting link came up on Hacker News tonight: http://news.ycombinator.com/item?id=2168332
This Microsoft paper seems to confirm that rather than just doing generalized learning on clickstream data, Microsoft has deliberately reverse engineered url parameters on Google to learn when Google is doing a spell correction. The paper seems to say that Microsoft was looking for specific Google url parameters like “&spell=1” that indicate a spell correction. Here’s how the paper puts it:
“In our experiments, we “reverse-engineer” the parameters from the URLs of these sessions, and deduce how each search engine encodes both a query and the fact that a user arrived at a URL by clicking on the spelling suggestion of the query
– an important indication that the spelling suggestion is desired. From these three months of query reformulation sessions, we extracted about 3 million query-correction pairs.”
Together with the experiment we recently ran, can you see where engineers at Google would be concerned? And would want more clarity from Microsoft about how clicks on Google and Google urls are used at Microsoft?
[Again, not speaking on behalf of Bing.]
(1) Was Google unaware that Microsoft was using clickstream data? Was it a surprise?
(2) Do you think it unethical that users, having opted in, provide the data to Microsoft?
(3) Do you think it unethical that Microsoft, having received these (public) data, would use them to improve the Bing search experience?
For what it’s worth, I think your answer would be yes to all three — but these seem surprising to me (except as posturing, which would also surprise me).
I understand the concern, but it seems to me more of an internal concern for Google; that is, an engineering challenge to provide a user search experience without revealing information publicly about what is happening under the covers. I don’t know anything about the research paper (other than what I read in the Hacker News article), but Google’s revealing when a spell correction takes place via a URL parameter seems to me to be fair game to note when building a data collector which feeds into whatever learning algorithm is used.
As to your second question — again, not speaking for Microsoft — I’m sure Google would like to know, but why would they be required to say? (In a related note, I didn’t quite understand the call by Harry Shum to create a joint standard for search quality. Isn’t this a decision best left up to the market?)
Will,
Your #3 glosses over the issue somewhat, IMHO. It would be more understandable if Bing had used this data as a learning experience to genuinely improve its search, by intuiting new ways to interpret user errors and designing better algorithms. But it appears that the reality of these “improvements” are closer to what most are calling them: straight-up copying.
It doesn’t look like Bing ever understood how or why Google returned the corrections it did, or developed ways to make similarly advanced logical leaps based on that data. It was more a matter of “Google returned site X for query Y, so we will too.” That’s not competition, that’s simple mimicry. Which explains why it happened even when Google’s corrections were flawed, in their own view. I guess it’s hard to do quality control on your competitors data when your dealing with three million+ terms.
As for the ethics of doing this: I suppose it’s technically allowable, since the user did opt in (though we all know that people rarely read EULAs or understand their implications). But I would hope that Bing would have the professional pride not to copy Google in such a transparent way. Learning from their harvested data and using it to engineer similar improvements, sure. But wholesale copying of millions of query/correction pairs? That’s just piggybacking on Google’s hard work (and it is hard work).
Hi Will,
Your defense of Microsoft and Bing would make more sense if these “synthetic” results were the only ones copied. If Google had set up these honeypots out of sheer paranoia, then it would be natural for Bing to pick up on some of them since there were literally no other signals to use. There would still be the issue of monitoring user interaction with Google to improve Bing, but at least it only would have happened in an isolated, artificial circumstance.
But according to Google’s own blog post, they did not think to set up this “sting operation” until it became clear over the course of several months that Bing’s top result was increasingly matching Google’s, for “popular queries, rare or unusual queries and misspelled queries,” as Singhal put it. Just look at the “torsoraphy” query, for example. Bing wasn’t recognizing the misspelling and correcting it on its own, like a competent search engine should — it was merely lifting the top result from the Google page for the same term.
If Google’s report is correct and this kind of mirroring was occurring “with increasing frequency for all kinds of queries […] Even search results that we would consider mistakes of our algorithms,” then that suggests that your take on the situation is inaccurate. Bing didn’t fall back on Google’s results for the honeypots alone as a last resort when all other search signals failed. Bing borrowed from them on a wide scale, even when they were flawed, to the point that Google noticed and decided to test for it.
As a heavy user of search, I’m a Google fan, but I recognize that Bing has innovated in some areas (image search, homepage backgrounds) and have used it myself on the few occasions when Google’s results are unsatisfactory. But I will use it less often, if at all, in the future. If Google can’t find it, why go somewhere that has to use Google as a crutch? And needless to say, I won’t be using Internet Explorer’s Suggested Sites or the Bing Toolbar anytime soon.
That said, I hope the folks at Bing do their best to walk this back and regain people’s trust. Microsoft has a history of leveraging their products in multiple areas to gain an unfair advantage over competitors — just look at Microsoft’s proposal to pay NewsCorp to delist itself from Google and give Bing exclusive search rights, or how the ubiquity of Windows made IE the dominant browser by default. This is just another sad example of Microsoft using unfair tactics instead of competing on merits — in this case, by leveraging their browser tools to crib notes from Google, a kind of obvious cheating Bing engineers should know to avoid.
Like Google, I want this practice to stop — not simply to protect their search from unfair copying, but so that Google has a genuine competitor. I’d rather have two (or more!) robust engines to turn to, a market incentivized to innovate and compete, rather than a stagnant field dominated by a single giant gone soft and a handful of competitors desperately aping their every move.
Interesting, Will.
So what I hear you saying is this. You have developed an algorithm that has, as one of it’s major inputs (must be major, or the impact wouldn’t be visible), a method for taking the output of our major competitor’s product, using the consumers of that product (users who elect to use google, not Bing, of course) as intelligent filters, and then adding those results into your results.
In effect, you are leveraging your desktop and IE footprint to turn IE users into intelligent filters on google’s output, and feed that input into Bing, so that Bing copies what’s good about good.
Algorithmically, that sounds great.
Ethically, doesn’t it sound a bit dodgy to you? It sounds like a crowdsourced version of “hire a bunch of people to sit in a room typing search terms into Google, and selecting the best matches to manually add into Bing.” This manual task would almost definitely be considered dodgy; so why is the aggregate, anonymous method defensible?
Blair,
(With reminders that although I work for Bing, I don’t at all speak for them).
I can’t speak to how major the impact is of the Google clickstream. It’s clearly important. As the link to my comment to Jordan states, we’d be foolish not to incorporate such data.
It’s clear to me that people don’t, and can’t understand all the implications of what it means to opt in. When people use Google’s search instead of Bing’s, they are not working for Google; they’re working for themselves. Having opted in, they also provide data to Bing which has helped Bing improve its search engine. No one is hurt; the data is made anonymous, etc., search in general gets better because competition drives improvement.
Personally, I wouldn’t have any ethical qualms of hiring people to do Google searches, and using the results to help train a selection or ranking model. It is, of course, too expensive to do such a thing. Of course, both Bing and Google do hire people to do searches on both search engines (and others, I imagine) in order to judge result quality. In the small, this is exactly what Google did in setting up the honeypot; I have no problem with their doing so (just with the conclusions that they draw). Again, I remain surprised that Google was surprised that Bing would be using clickstream data; I’m still trying to wrap my head around that.
“Personally, I wouldn’t have any ethical qualms of hiring people to do Google searches, and using the results to help train a selection or ranking model.”
Wow. That’s shocking.
Why, Blair?
Why? I think it was pretty clear above that I think it’s unethical. So, I find it shocking that someone who works with the technology, in the area, sees no issues with it.
Stating it is unethical isn’t the same as it’s being unethical. I’m seriously asking you: what creates your shock? Where are the ethical lapses?
As a Googler who is not speaking for Google, I don’t see any obvious ethical issue with Bing’s approach.
I also don’t see a copyright issue, or even a violation of Google’s terms of service.
The outcome (Bing returning a result only because Google also returns it) could be achieved by illicit means, but Bing isn’t doing that. Either people are confused on that point, or they believe the outcome itself is wrong no matter how it is achieved.
Well, there you have it, a Googler has spoke. Seriously, if you guys don’t see the issue, I’m not going to be able to educate you, and I’ve devote enough keystrokes to this. Perhaps you’ve been stuck inside your little technocentric universe too long, if you can’t step back and see it.
Jordan,
Here’s a very cogent technical description, by a former Google engineer, of what likely happened:
http://www.quora.com/Did-Bing-intentionally-copy-Googles-search-results
I love the quotes in that article you point to, Will: “For Bing to use click and visit data from a competitor seems to be more clever than wrong given their lower market share and lower organic quality” and “All of these steps are really logical extensions of the previous steps, and ignoring the wealth of available data if they provide good quality signals would have been silly.”
Which is to say “Stealing is ok if you are the underdog” and “Obviously, if you can steal your competitors data, and it would make your product better, not doing so would be silly.”
Huh? Just because it’s possible, and algorithmically advantageous, does NOT mean it’s ethical or legal. That’s why we have privacy standards and laws, and things like copyright and patent systems. Right?
Blair, Ah, we are cross-posting.
The data that Google provides are public by the nature of the service. Google returns results on a SERP (the search engine results page) for query X in A,B,C order. Users, opting in to forward that data (under the opt-in agreement) are providing Bing public information.
The point of being the underdog is this: because Google has been a better search engine (“lower organic quality,” as measured by the kind of human judgments I mentioned in the previous reply), it makes sense to use the data to improve our search engine in a way that the reverse might not.
I find both of these assertions absurd.
The data is publicly visible; that doesn’t make it public, nor does it make it ethical for a competitor to copy it, or reverse engineer it. Putting a user’s keystrokes in the middle doesn’t change anything; it just makes it more computationally tractable. By your argument, it would be reasonable for Bing to have an automated system that takes all queries to Bing, sends them through a virtual keyboard into Google, scrapes that content out of the resulting HTML of the SERP, and feeds it both into Bing and back to the user.
Your argument about being the underdog completely misses the point: OF COURSE it makes sense for Bing to WANT to use Googles data and for Google not to want to use inferior data. Just like it makes sense for any company to steal from their better and larger competitor. I won’t even expound upon the irony of someone at MS making this argument. :)
Pingback: Google-Bing Controversy (Feb 2011) Bing Side of the Stories | The Bento Box
Will,
Bing should give some credit to Google after they have worked real hard for their *Conclusive* evidence. Why not add a sort of disclaimer at the top of the Bing SERP like… “We acknowledge Google as the sole provider of results for Nonsensical and useless search terms”. That should keep them proud and happy!
Pingback: What Problems Search Engine Algorithms Have To Tackle? | Bits Of Knowledge
Will,
After I commented on one of your later posts, I took a look at this one.
Here, referring to the Wikipedia article, you say that a clickstream “is the record of searches, displayed results, and selected results”.
That’s not what I get from the Wikipedia article, or from my general impression of clickstreams. If your clickstream includes displayed results from Google, you have scraped Google. That’s more troubling than just using the user’s selection.