Analysts firms vary massively in their independence. That’s one finding of our 2013 Analyst Value Survey, which shows the opinions of 352 users of analyst research.
Ovum, NelsonHall and Redmonk stand out as firms widely seen as highly independent, with strong scores for 451 Group, Pierre Audoin Consultants, Saugatuck and TBR.
Perhaps unsurprisingly, many of the firms that are most seen as independent are also seen as the least independent, which draws down firms like Gartner, Forrester and IDC in our chart. Thanks go to Ludovic Leforestier, who suggested this way of charting the data for us last year. It’s taken a while to get around to it; we’ve been spurred on by the forthcoming IIAR forum discussion on ethics.
No study is perfect, and it’s worth making some points about this chart. The more respondents scored a firm as being independent or not independent of its commercial relationships, the further its data point is over to the right. Don’t pay too much attention to the data over on the left: it’s there for entertainment value only, and we’ve blanked out a few firms where the number of responses was low.
The higher up on the chart a firm is, the more it’s seen, on balance, as producing research that’s independent of its commercial relationships. We’ve shown that number as the percentage of people scoring that firm as independent, divided by the total of the numbers that scored in either independent or not independent. {For example, Gartner was selected as most independent by 67 people but as least independent by 62. Gartner gets a score of 52% [that is 67/(67+62)] and is the furthest to the right because the total number of people selecting it in either of those questions [129, which is 67 plus 62] was the highest out of the firms in the survey.}
One of the options was “Other…”, which allowed people to write in the names of smaller firms. It’s the data point without a label that’s between Everest and IDC. There were lots of them and, as you can see, as a group they are averagely independent. That point is unlabelled, since it doesn’t add much.
The black line rising across the middle of the chart is a trend line automatically added by Excel. It suggests that the more people there are commenting on a firm, that the more independent it is. Perhaps that means that the more eyes there are on a firm, the core careful it is about its independence.
Some of the firm had scores that were so close that the labels went on top of each other. So that the names are legible, I’d dragged a few down so the labels underneath can be seen. I’ve done that by picking up the label that’s longer (since the shorter one is hidden under it) and pulling it down a little. When you see labels that are very close, that’s what happening. So, for example, Saugatuck and TBR had essentially identical scores. So did GigaOM and Current Analysis. BARC and Canalys were also evenly matched… you get the picture. Where there’s one dot and more than one label, their dots are on top of each other. One day we’ll all have 3D monitors.
We’ll conduct this survey again in September: with more responses, we’ll have a more accurate picture of users’ perception of analyst independence.
P.S. One useful caveat about this chart came from an analyst who gave feedback on it: this chart is “all about perception”. Needless to say, that is exactly what surveys do show. Analysts’ clients are the people who are best able to make an informed judgement about the relative independence of these leading analyst firms, and there’s no more effective alternative to a survey. We don’t want to be complacent about the research, so we remain open for suggestions on how to improve the study.
Dunc,
I really don’t get your charts: to me it seems you’re plotting the same things on Y and X. It would be much more useful to estimate the % of vendor revenues on one axis and perception on the other. Or, better, the largest client as % of total revenues.
That would be much more useful….
PS: what’s the distribution of clients of those firms and is there correlation between what firms they select as more independent?
Hi Ludovic. I’ve added an explanatory comment {in curly brackets} about the X and Y. There are two sets of numbers used to produce the charts: one series is firms selected as most independent, and the other shows those selected as the least independent.
The horizontal axis shows the sum of those two numbers, and shows how much data we have about that firm. The further right that firm is, the more confident we can be that we are expressing the views of the respondents because there are more scores for Gartner, for example, than about EMA.
The vertical axis shows the ratio between those numbers: is the firm seem more as independent (higher) or now (lower).
So, yes these are the same numbers but the axes show different things: one shows the volume of scores relating to each firm, and the other shows whether each firm is seen as more or less independent.
We have compared influence on buyers versus suppliers (See slide 10 in the 2013 Analyst Value Survey – summary slides presented at the Analyst Relations Forum at http://www.slideshare.net/dchapple/13-10-analyst-value-survey ) but of course a survey of end users cannot show us the revenue of analyst firms.
PS I don’t get your point about the distribution and correlation. Correlation with geographical distribution, or some other distribution?
I understand what you’re trying to show here Duncan and I sympathise with how difficult it is to achieve any measure like this. But here’s some more thoughts on independence for you:
1. It’s impossible to measure and isn’t clearly defined.
2. It varies by analyst – there are some analysts who stand firm within less scrupulously independent firms, and some analysts within so-called independent firms who are plainly bought and paid for. I’ve seen analysts within certain high scoring firms on your chart whose output is cut and pasted from vendor decks I’ve viewed at the same time as them. I’ve, also heard them present as “their ideas” messaging directly from a vendor. I don’t think they always do this on purpose – it can be a subconscious process of indoctrination.
3. Independence is a marketable asset – half the perception is not factual but based on some firms marketing their so-called “independence”. Some may be fiercely independent but just don’t make a song and dance about it. Having an independence charter, framework, process etc only works if it’s actively enforced – often it isn’t, sometimes it’s deliberately subverted, sometimes it’s only marketing. As more and more analysts become salespeople for their research, I find it hard to believe the strong line between research and business exists in many firms.
4. Perception is trailing fact. Some firms independence varies considerably over time. However due to historic independence (or lack thereof), combined with factor 2 above, some firms retain an aura of independence (or lack thereof) long after they sold out (or got their house in order). It just becomes a fact that is repeated long after it ceases to be true.
5. Independence can vary according to perspective. Some firms only work with larger vendors. They may treat these vendors in an even manner, but they exclude smaller vendors because these vendors can’t pay for their services. This is an approach that makes me personally very uncomfortable. It creates inertia and favours larger vendors, so to me isn’t true independence. Another factor about perspective is that people tend to believe (and say something is independent) what they want to. I’ve heard “and they’re independent” repeated so often as a justification for a position. Being independent doesn’t make you right. It’s possible to be independently stupid, for example. Is that better than being biased but clever? Vendors cultivate a perception of independence while then attempting to undermine it/manipulate it for their own purposes. They also big up certain analysts because these are the ones that agree with their position.
6. Many of the most independent analysts feel they can’t be independent within a firm, so they’ve gone out on their own and become ‘independent analysts’. Your study doesn’t really capture these guys, who within the market they work might be highly valued, highly independent and very influential. Some firms have a corporate position, and they don’t brook analysts who go against that. Hence why many truly independently minded people become independent in terms of how they go to market.
7. Ultimately we could argue that independence is illusionary – we’re all biased, that’s the nature of being human. Independence is about trying to create a structure to counteract the tendency for us to favour certain things over others and give a fair hearing (that could be a mental structure or a process). For me independence is not just about favouring one vendor over another, or excluding certain vendors etc etc it also applies to approach and technology. I find some analysts become “stuck in their ways” and favour certain technologies and technical approaches simply because they understand them or made their careers on them, and don’t look at new ways of doing things or disruptive technologies. When they do, it’s only because the technology has become mainstream and is threatening their business. I get a lot of complaints from disruptive innovators about this effect that analysts have and in this sense an analyst can be badged as independent while actually advocating a whole part of the industry against another (emerging) part because it benefits their career more. Human nature is that some people hate to admit they were wrong or to say “forget that, now there’s something better”.
8. The need to please or be nice is not a good characteristic in an independent analyst. For me independence is about thinking independently – being able to see new ideas, evaluate them fairly and effectively, and to the best of our ability come up with effective strategies or tactics as a result. That means risking ridicule, controversy and opposition, and not just going with the flow. It means saying “no”. It means being the one to stand up in a room and tell someone their great idea (their baby) is ugly. That’s important because the “yes men” analysts are not independent thinkers. They do an awful lot of harm. Sometimes a company really doesn’t want to hear the truth, but needs to. The bravery to be a truth teller is a rare commodity.
9. Independence isn’t creativity – and the latter is even rarer than the former. Going back to my earlier point about insight and intelligence – how do we value this characteristic in balance with others?
I think if you asked analysts if they agreed with the above survey there’d be some chuckles. We all know or suspect the truth at an analyst level.
Thanks for this Teresa. I’ve replied in a separate post to draw attention to your comments.
Fantastic comments from Teresa -indeed!
Dunc, on your questions:
1/ You’re still comparing two sides of the same thing. The chart is like a MQ (Magic Quadrant) of creme paints versus off-white tones. In the end, it’s still about the same thing. As I said, doing some research on factors impacting independence or actual vendor coverage and bias by each analyst firm would be much more interesting.
2/ What’s the distribution of respondents according to their patronage of the analysis firms they rate?
3/ My second question is directly linked: my hypothesis is that one tend to be positive about what one buys -call that self-fulfilling or reinforcing. So if you buy, say IDC, you’ll say they’re independent, right? Just like iPhone users would say they’re the best device even if they have never tried anything else.
Hi Ludovic. If you have any suggestions about how someone could do that research, then let’s have a go. The survey doesn’t ask about patronage, and I’m not sure that most users really know which analyst firms their company does, or does not, pay.
That’s why it would be interesting !
I’ve added that question into the survey for H2 2014, by the way, and the results are – as you predicted – interesting. They show the power of freemium.
Good discussion and great comments by Teresa and Ludovic – I particularly like the thread that vendor independence does not correlate necessarily to an analyst’s ability to provide valuable or creative insights to clients.
To add to the discussion about perception: I have noticed that the perception of the analyst firm is highly correlated to how the firm ranks the vendor. Especially if a vendor is ranked pretty low, they make the most noise about the independence of the analysis instead of the reasons behind the ranking. Additionally, I am not sure how the market values independence. Would consumers of the research be willing to pay a 50% premium for an independent analysis (their perception)? A 10% premium? How would they measure or validate independence? My guess is that since everyone is used to getting information for free that they are not willing to pay a big premium and they just evaluate everything they read with healthy dose of skepticism.
David’s right that vendors complain about independence when they don’t do well. I doubt that effects our survey, since the different outcomes for different vendors will tend to cancel each other out, and will be muffled by the end-user participants in the survey. My feeling is that average contract valued increase as we go towards the top right, so there is probably a correlation between independence and price.
[…] Founder & Chief Strategist at Telesperience, had made some very useful criticism and comment (here, and well worth a read) about our recent chart of the Analyst Value Survey data on analyst […]
I am going to take the fact that my firm is not listed as an indication that it remains somewhat stealth even though it dominates certain segments of the networks market. It is extremely easy to pick off accounts from those firms who are listed given their value proposition potential lack of data.
@ddines OEMs will always not like you as an analyst if you rank them lower. However that doesn’t mean they won’y but from you.
@Teresa i think you are spot on for all your comments
Whitepapers are a key indicator of “independence” given how much they cost to publish and who the sponsors are.
How about a reliability index? Is the information from the analyst reliable or not and can you trust them? You can’t rate a firm like this but it would be like the Wall Street All Stars for financial analysts. There are always stars within each firm but the overall firm may suck. The value of an analyst isn’t always in what is published or written I can tell you that.
[…] is one of the firms they have used: that puts it in the top 20 firms worldwide, on a par with TBR and Digital Clarity Group. Many of Constellation’s competitors are uncomfortable with […]