Lighthouse considers itself to be independent from the analyst firms. Every year or so an analyst will ask if we are, sometimes directly or more often not. It’s almost always a response to the same stimulus: our annual survey of vendor-side users of analysts’ services.
Last week a head of research at one analyst firm sent us a critical note, writing that “These may sound like sour grapes, but I assure you they are not (could be grape influenced, though).” Reflecting on our newsletter, which recently carried an item headlined Vendors name rising analysts: Forrester, Gartner, Ovum, Aberdeen and Burton, the director comments on three aspects.
- That it gave the impression of being sponsored by an analyst firm.
- That our mini-survey on influencers participants compares apples and pears. For example, the different segments of technology are very different. An analyst firm can be unknown in one market, yet in another regarded in one market as pre-eminent (not just in thought leadership and quality of market research, but as “the people who know” by leading thinkers).
- That one of the names on the list has absolutely no real “influence” at all. In their opinion a small number of execs about such diverse areas is meaningless.
Our comments are these:
- The survey is not sponsored by anyone. It’s a free annual survey. Participants donate the time to contribute data: all they get is the results. The labour to analyse the results is donated; we don’t charge for the results. We don’t see why anyone would think it’s sponsored, but we’d like to have some suggestions. For the record, our research is sponsored only by vendors. We do not work for analyst firms at all, other than giving them our view of their market.
- The survey does not compare apples and oranges: it compares firms that are big with firms that are small. A small firm probably won’t win as much influence in a year as a large firm. That is real. In our opinion, it’s the right way to show the data. There is an option, which we dismissed. The option is that instead at looking at the changes from year to year, we could look at the rate of change. For example: firm A goes from influencing 5% of firms to influencing 10% of firms; but firm B goes from influencing 0.05% of firms to influencing 0.15%. Our current method makes it look as if as firm A has increased the most. We think that’s right. The alternative would be to focus on the fact that Firm B has had a larger percentage growth, but it’s still tiny. We think it would be much more misleading to say that firm B has increased its influence the most. That’s our approach, and we stick by it.
- Our survey asked vendors’ staff what they thought about the analyst firms that they consider to be influential. Every year we say the same thing to participants: “Of course, the usual health warning applies: perception is not reality.” Just because vendors think one firm is more influential than another, that doesn’t mean they are right: but it does mean that this is what they think. However, we do not think these data are meaningless. Our survey is aimed at analyst relations managers in the 100 largest firms; to get more than 50 participants gives us an excellent sample size. The data are also rather stable; different people have similar opinions. We think this survey accurately shows what those people think. They might be right or wrong, but one cannot dispute that these are really their opinions.
Of course, we have invited this head of research to give us call to discuss our methodology in depth. As former analysts, it’s something we look forward to.