Is there a risk that role-based analyst services can contradict the research process? One of META Group’s weaknesses, which was a strength in the eyes of some, was the fact that its devoted alignment to its clients’ preoccupations often took precedence over the research process. I’m staring to wonder if the same risk might mature in the role-based services offered by Forrester or Gartner.
It’s clear that many clients of analyst firms are delighted by the development of role-based approaches, especially because of the sharp contrast between the approaches taken. Forrester and other participants in the market are taking approaches that differ from one another, which expands the choice available.
Of course, I will be interested to see how Gartner and Forrester change as they become more rooted in the role-based approach. Clients will become increasingly comfortable with analysts climbing onto ‘their side of the table’. However, META Group aligned to its clients so strongly that it showed a tendency to ‘overfit‘ the attributes of a non-random sample; its customers. Forrester and Gartner are a long way from that point, of course. Indeed, one could say that this would be a better problem to have than being marginal to clients’ needs.
I am starting to see some examples of the dynamic that e certainly saw unfold at META, and which could unfold elsewhere. Their market rewarded them for overfitting, and that delight provides a material and relational force that needs to be assessed. It sets up dynamics, and imbalance, that can only be managed consciously. Happily, almost everything that analysts say to me about this risk helps me feel that role-base services services are developing the right values, culture and technique to avoid META’s experience.
In a way it’s little unfortunate that ARmadgeddon’s recent discussion about Forrester’s AR services stands outside that discussion. The issues that Forrester may encounter are general illustrations of what happens when role-based services are driven by their immediate stakeholders rather than the wider market.
In Forrester’s case, by basing its research on a small sample of analysts who principally work at Forrester, its AR service can make recommendations that don’t reflect the objective best interests of analyst relations managers. Forrester wants to develop some original insights into the issues of AR managers, and so it doesn’t want to repeat the findings of others. But, for example, it’s ineffective for to focus on Forrester analysts’ stated preferences as if that’s the best way to run AR. Analysts want a qualitative increase in candour and resource in analyst relations programmes, and that won’t happen. Therefore the task is to identify what the priorities and trades-offs are that allow AR programmes to best meet analysts needs. One can imaging that similar tensions can build up with other role-based services, at any firm.
These comment about overfitting should not be seen as a specific criticism of the work that Forester’s AR team is doing, and it should not be seen in that way. Their clients are largely delighted with the role-based services, and so they should be. However, unless role-based services use the same broad process of gathering data that analysts would normally aim to use then they will reach analysts that fit their unrepresentative sample, but may not reflect the wider realities.
1 Comment