The IIAR recently held one of its most remarkable webinars, with Outsell VP, Harry Henry (@harryhenry). A post about the webinar explains that “Harry Henry leads Outsell’s research and advisory programs in the areas of marketing, media and analytics”. He’s one of the most experienced market researchers in the field, and gave a fascinating and candid presentation. IIAR members who missed the call should check it out on Huddle.

The webinar was a substantial opportunity for AR professionals. Outsell is a highly professional market research organisation. Their analysts are best known for tracking market share for the information industry, which we’ve been following since 2006. Their data has been vital for clients of ours looking to refuse the every-present myth that analysts’ influence is declining. Furthermore, their qualitative ‘local knowledge’ of the industry is remarkable. As we noted in other posts, they track, and repost, material of ours (even without telling us but, hey, we love the traffic). They even produced a great guide to analyst relations consultancies.

That said, we’ve also had some concerns about the firm’s methodology, which we noted back in 2006. A glance at the leader quadrant above, which shows Outsell’s view of the industry in 2011, will give most people familiar with IT research some cause for thought.

As with most issues with strong research methods, the weaknesses of Outsell’s approach are generally the unavoidable consequence of its strength. Outsell has to tabulate its attributes at the level of granularity that allows IT researchers to be rolled up and compared with other market research firms, and it puts into that segment firms as different as the Corporate Executive Board and NICE Systems. Furthermore, those firms have to be rolled up with the whole information industry. That is a huge task which I certainly don’t envy: no-one does it better than Outsell.

That said, there are six obvious weaknesses.

1. – One issue that that the attributes it collects have to be available, not too onerous to collect and comparable across market segments. For example, a key factor for many AR professionals is to understand where the revenue of analyst firms comes from and, in particular, how much of that revenue is coming from buyers, such as enterprises and carriers. Outsell doesn’t provide that data.
2. – Also problematic is the way in which firms are allocated into segments. How meaningful is it to NICE Systems to be thought of as a market research firm and compared to CEB? Is that, actually, useful to anyone? It’s not exactly a game of any pig in a poke but it’s very tricky to have to place each firm into one segment out of a small set.
3. – Flowing from that, the resulting issue is that firms are not split easily across segments: Outsell’s list of the top IT research firms is a list of firms which are better described as IT research firms than anything else. They are more specialists in that than any other field. However, it excludes generalist research firms that can be major players. As the 2011 diagram above usefully suggests, Neilsen and GfK must surely still be among the ten firms conducting IT market research, but when the top 10 IT research providers were presented at the IIAR call, the list comprised only specialists.
4. – Another issue is that the Outsell Valuation Model and its research process reinforce each other. The valuation model is a hugely useful tool and one that they should be proud of. Their approach combines the data they collect to allow information providers to model their market value. It’s hard to change the model in ways that require data that the research process does not gather, and its hard to change the research process away from the approach that currently supports the model. That produces a lack of adaptability in the model (and, honestly, the resulting consistency is probably more highly valued by Outsell’s customers).
5. – The flip side is that the the valuation model doesn’t focus on value to customers, but to investors. Our Analyst Value Survey is a very, very modest tool. If the Outsell research process is like China, our survey is like Andorra. There is no comparison. But, the difference is that we are tracking which activities are specifically generating value to users, and which activities are most highly valued by them. Outsell’s method allocates all of Gartner’s revenues, even the ones which are not specifically revenues from information services, to the attributes it follows. Consulting and events are counted into the totals for research and trading services. That means that it’s not breaking out the specific elements, and we can’t see the value generated by events as events, or by consulting as consulting. Nor did we see other providers of those non-research services in the IT listing; we might have expected to see Informa, for example, which organises a lot of events and conducts a lot of research. If Gartner moves up the ranking because of Symposium, why isn’t Deutsche Messe there with CeBIT?
6. – That said, the approach means that the model and other outputs rests on the metrics that are collected and not on the ones which are not. That is where we come back to the diagram above. Consider the placement of Info-Tech. There is no meaningful sense in which Info-Tech is as able to execute as Gartner, or more in tune with tomorrow than any other analyst firm. That is simply silly, and anyone in the industry other than Info-Tech will surely be as baffled. That finding is quite impossible to reproduce by any other credible method.

There’s more we can say about what Outsell had to say, and we’ll share that in a week or so pending some diplomatic discussions with friends in the IIAR.

But our verdict is pretty clear: misadventure. Outsell’s approach is one-sided in that it is totally fit for its intended purpose (producing comparable market size date which can drive a valuation model), but quite unfit for describing the IT analyst industry in a useful and reliable way. At this low level of granularity, the Outsell data are just too lumpy and inexact to give a true picture of the market.

Needless to say, the blame for this rests fully with me and other IT industry professionals. As all those links at the start of this article show, I and others have a voracious appetite for Outsell’s data. If they did not pull and push it in cruel and unusual ways, others would. But it would lead to similar failures, or worse. Outsell cannot possibly justify the added cost to develop a more granular collection of data unless someone like Gartner’s CFO commissions it (hint, hint) and until that happens, we just have to trust Outsell’s substantial qualitative insight and not ask its to present its data in ways that conceal as much as they reveal.

PS An different, earlier, version of this article is available in the IIAR’s Huddle page.