The new ‘Analyst Direct‘ service has serious flaws. Many of these issues relate to its market intelligence engine, ‘MI Analyst’. Some of the issues I mentioned elsewhere have been resolved: for example, ‘Ph.D.’ was listed as an analyst. However, three principal issues remain.
- Some issues reflect data quality.
- Some of them are statistical issues, rooted in the Numerophobia which is found in some parts of the AR community.
- Finally, the reach of the database is short of the 95% coverage the service claims.
Data Quality.
As a beta-tester of the service, I raised a number of issues with Northern Light, some of which are illustrated on the screen shot on the right.
Among the ‘companies’ listed are ‘Philippines’ and ‘AG’ (the German equivialent of Inc., Corp. or PLC.) I’ve also seen ‘Publications’ listed as a company. As you can see, the system has a major glitch with firms whose names include the name of another brand: There’s no listing for ‘Alcatel-Lucent’, for example, only separate entries for Alcatel and Lucent. That’s bad news for Nokia Siemens Networks and Fujitsu Siemens Computers: and gives misleading data to their parent firms.
Of course, such issues may say something about the testing of the system, the analytic power of the machine intelligence in the system, and the resources available to maintain it.
Statistics.
Another major issue with the way that ‘Analyst Direct’ works is the way in which it charts the changes in share of voice data. The screenshot below shows Analyst Direct’s way to charts the share of voice for the ten firms most relevant for telecoms. You can click on in to enlarge.
There are some clear issues here. There are almost 500% more data points for 2007 as for 2006. Almost two-thirds of the data come from one firm: Current Analysis. Gartner gets 5% of the 2007 score. Yankee Group fails to get into the top ten (it is 13th out of 16 firms).
Most people with any familiarity with analysts following the telecoms industry will know that what Analyst Direct is showing is not useful information. What’s going on in this chart is that Analyst Direct is counting each item from Current Analysis as if it is the same as one item from Gartner or Forrester. Of course, Current Analysis is a small firm that produces many small items. Gartner, Forrester and Yankee produce fewer items but each of more influence.
The inability of the system to scale the data means that firms that produce normal volumes of research get swamped. For example, Analyst Direct recently expanded the number of firms by one-third but, taking the sample of reports on software, that one-third counts for around 11 percent of the sample, while Current Analysis is 32% of the sample.
Northern Light may face some difficulties in resolving some of these issues. We know from our own experience there’s a simple way to get around the problems of firms that produce different volumes: the solution is to use statistics like standard scores to adjust each data series so the results become comparable. However, some analyst relations advisors seem to be scared of statistics. For example, KCG’s newsletter has described statisticians working with share of voice data to be “snake oil merchants” – as if using Z scores to make data comparable is, in some way, “a form of tampering with data”. Since KCG is the principal reseller of Analyst Direct, Northern Light may struggle to convince its stakeholders to support them in adopting statistics.
Reach of the database.
Northern Light claims that “Analyst Direct now enables users to conduct integrated searches of 95 percent of the billion-dollar database of all IT industry analyst research published annually”. In fact the tech analyst industry has been sized at anything from $2.5bn to over $10bn by a range of researchers. Out of 750 analyst firms, Analyst Direct overs just 19, two of which are headquartered outside the US. Almost 100% of the research is in English, other than a few papers by Celent in French.
There are also major issues with the ability of the system to track all the research published by those 19 houses. For example, search for Treasury within Gartner research over the last 12 months: Analyst Direct gives four of the 65 items shown on Gartner.com.
As a simple data integration tool, Analyst Direct is very promising. However, as management information tool it is immature: it will prevent some vendors from getting a real focus on which firms are mentioned the most in influential research; it will encourage vendors to spend more time with analysts who produce volume rather than quality; it will turn AR teams away from international markets.
Of course, all of that produces great opportunities for those AR teams who can identify the analysts who are most heavily influencing sales. While their competitors are wooing Current Analysis, they can focus on the more influential analysts.
P.S. Thanks to Sherry for her comment wondering if we have a competitive product to Analyst Direct which is motivating our concerns. We don’t have a similar product, and we’re not aimed at the same market segments as Analyst Direct/KCG. Our Analyst Index and Analyst Mindshare Benchmark services also use share of voice metrics however, so we do have deep expertise in the area. Not that you’d need deep expertise to be worried about these issues with Analyst Direct.
