Benchmarking - KDD and Netflix

Machine learning and statistical methods are at the heart of what we do. They absolutely are. This is because we’re striving for the most accurate answers, with a latency that makes them commercially valuable. Creating the best engineered solution that delivers undisputed value is what we are about. In our world if it doesn’t provide a measurable performance improvement for the task at hand then it isn’t useful.

We apply the discipline of benchmarking in almost everything we do (...it may not surprise you that we operate as a true meritocracy at Causata... I absolutely believe that if somebody can do my job better than me then they should be doing it...!!) It keeps us focused and real. I recall some early days in 1997 and 1998 when I was privileged to work with two exceptional database marketing practitioners, Jacob Zahavi and Nissan Levin. They wrote some software at Urban Science that won the KDD cup for two years running. A significant difference between their approach and their more academic competitors was that these guys ruthlessly benchmarked everything they did against previous work, using large representative data sets. This prevented unfounded preferences in their methods arising through familiarity, discouraged misplaced pride, and encouraged the exploration of new techniques.

The frantic developments leading up to the Netflix prize on 26th July 2009 were a very public demonstration of the power of transparent benchmarking in developing engineering solutions for decision science. The ranking of performance clearly accelerated the validation of approaches that worked well and helped identify new hybrid opportunities.

It is huge fun working with talented and energized people in developing new solutions, but I have learned to respect benchmarking and champion challenge as processes that keep us grounded and drive engineering progress.

Paul.