In an episode of The Simpsons when Homer creates a vigilante group to track down a cat burglar, Lisa, concerned over their abuse of power asks: “If you’re the police, who will police the police?” to which Homer replies: “I dunno, Coast Guard?”
Lisa’s question is that of Juvenal: qui custodiet ipsos custodes? And I am reminded of that by the news that AllTheAnalysts.com now allows users to score analyst research found by the Google search engine.
It’s not a new idea. If you look at Forrester.com you’ll see a similar system. However, on the Forrester site, which has a lot of users, the actual number of scores is pretty few. Not many people are volunteering scores, even though those users have access to the full text. On may papers, just one or two votes sets a very high or low score on reports (but perhaps that’s a starting-off sort of problem).
My bet is that the user-scoring of AllTheAnalysts will develop a systematic bias. The system identifies research, but users can click a button to see only free research. That free research will not only get more readers, it will also produce a tendency for readers to score free research more highly (since they can read the free stuff, but not the subscription content). In this system, free research will get a high score and paid-for research will get a low score.
But there’s a further bias: what sort of research is free? Often it’s vendor-sponsored research, either hosted on vendors websites or vendors have funded the research, so the reports are free on the analysts’ websites. The end result may be that AllTheAnalysts becomes a method for vendors to push their point of view into the market, with the gloss of web 2.0 community.
In the meanwhile, I am just waiting for free research firms to get higher scores, and then start to claim they have higher quality.