This week saw the 16th conference on Knowledge Discovery and Data Mining held in Washington DC (KDD-2010). “Why would we be interested in such an obscure conference?” you may ask. Well the answer is simple, Google presented their infrastructure behind how they test changes to their user interface, algorithm and methods content presentation.
Google essentially stated that they are always testing in fact they are tweaking things on almost a bi-daily basis. According to this Wired Story (How Google’s Algorithm Rules The Web) Google are likely to do over 550 tweaks in 2010, with Matt Cutts in one of his recently blogs saying that there will be at least a change a day to the algorithm alone. So I guess that’ll keep the Tamarites on their toes!
So what can we learn from all of this?
“At Google, experimentation is practically a mantra; we evaluate almost every change that potentially affects what our users experience. Such changes include not only obvious user-visible changes such as modifications to a user interface, but also more subtle changes such as different machine learning algorithms that might affect ranking or content selection…” (Google’s testing mantra)
The excerpt above is taken from Google’s Overlapping Experiment Infrastructure: More, Better, Faster Experimentation paper. This outlines concisely how Google performs tests and implements changes. To cut through the chaff I will try and summarise the process in more layman like terms.
Essentially Google layers tests into their results, page designs and into their user interfaces. Us unsuspecting Lab Rats are being tested and potentially tested for many different things simultaneously. Google layers into the resolved results pages its tests. When you make a search, the results are collected from the data-centres, compiled, filtered and then displayed:
Google adds to this a layer of tests which it controls, not everyone will be subjected to a test, solely a sample set. If you are part of that set your cookie will hold the test results and feeds them back to Google for evaluation. You may have noticed tests happening, I certainly have – one minute you see one thing the next minute you don’t, apparently according to Google this is always happening.
So what does this mean for us?
This essentially means each search even with the same keywords could generate an entirely different set of results if an algorithmic test is in play. Different users, browsers, timings, locations, keywords, could all bring about completely differing versions of Google results. It is important for us to keep on top of these tests, yet also not read too much into them if they are positive or even negative, as after all it’s only a test!
Google do labour the point that they do not just roll out tests to all results, they are very scientific and controlled with their methodology. If a test does bring the results they want, only then do they roll in the update to their algorithm.
As always we like to keep a keen eye on these tests as after all it could show us the key to Google’s thought process and future developments. I never thought I would say this but, I for one am happy to be a Google Lab Rat!Tweet