Our Analyst Value Survey (AVS) considers itself to be independent from the analyst firms. Every year or so an analyst will ask if we are, sometimes directly or more often not. It’s almost always a response to the same stimulus: our annual survey of users of analysts’ services and insights: a large majority of the survey participants work for vendors or service providers

Last week a head of research at one analyst firm sent us a critical note, writing that “These may sound like sour grapes, but I assure you they are not (could be grape influenced, though).” Reflecting on our newsletter, which recently carried an item headlined Vendors name rising analysts: Forrester, Gartner, Ovum, Aberdeen and Burton, the director commented on three aspects.

1. That it gave the impression of being sponsored by an analyst firm.
2. That, in our survey on influencers, participants compare apples and pears. For example, the different segments of technology are very different. An analyst firm can be unknown in one market, yet in another regarded in one market as pre-eminent (not just in thought leadership and quality of market research, but also known as “the people who know” by leading thinkers).
3. That one of the names on the list has absolutely no real “influence” at all. In their opinion a small number of execs about such diverse areas is meaningless.

Our response to these comments are these:

1. The survey is not sponsored by anyone (if the AVS ever has sponsors, that connection will be public and won’t impact the survey design). It’s a free annual survey (organized by my colleagues and I in time donated by our company). Participants donate the time to contribute data: all they get is access to a summary of the results. The labour to analyse the results is donated; we don’t charge for the results summary. We don’t see why anyone would think it’s sponsored, but we’d like to have some suggestions. For the record, our other research is sponsored only by vendors (most obviously the Analyst Attitude Survey). We do not work for analyst firms at all, other than giving them our view of their market (tiny engagements where they pay a token amount for travel and time preparing a presentation).
2. The survey does not compare apples and oranges: it compares firms that are big with firms that are small. A small firm probably won’t win as much influence in a year as a large firm. That is real. In our opinion, it’s the right way to show the data. There is an option, which we dismissed: The option is that instead at looking at the absolute changes from year to year, we could look at the rate of change. For example: firm A goes from influencing 5% of firms to influencing 10% of firms; but firm B goes from influencing 0.05% of firms to influencing 0.15%. Our current method makes it look as if as firm A has increased the most. We think that’s right. The alternative would be to focus on the fact that Firm B has had a larger percentage growth, but it’s still tiny. We think it would be much more misleading to say that firm B has increased its influence the most. That’s our approach, and we stick by it.
3. Our survey asks (mostly) vendors’ staff what they thought about the analyst firms that they consider to be influential. Every year in our summary we say the same thing to participants: “Of course, the usual health warning applies: perception is not reality.” Just because vendors think one firm is more influential than another, that doesn’t mean they are right: but it does mean that this is what they think. However, we do not think these data are meaningless. Our survey is particularly aimed at analyst relations managers in the 100 largest firms; to get more than 50 of those participants gives us an excellent sample size in our professional community. The data are also rather stable; different people have similar opinions. We think this survey accurately shows what those people think. They might be right or wrong, but one cannot dispute that these are really their opinions.

Of course, we have invited this head of research to give us call to discuss our methodology in depth. As former analysts, it’s something we look forward to.