Avoidance, bias and insularity are top challenges with AR measurement

What are the top errors made by AR teams trying to develop measurements and integrate them into balanced scorecards? Carter and Dave have some suggestions, but while they certainly picks up on challenging aspects of measurement, I’m not sure if they are all mistakes – let alone the most serious mistakes.

To pick up on just two of their ‘top five’:

  • Personally, I rarely come across AR managers trying to shift metrics they can’t affect. Sometimes you see teams that aspire to measure things that are hard to measure (for example, sales) but they are not being measured.
  • Just because data are granular, that doesn’t mean they are hard to collect. Surveys of analysts are much harder than tracking highly-granular items of reports, citations and so on. It’s much more common to find metrics methods that are hugely under-laboured, and are to simple to give guidance.

Much more major problems are

  1. Avoiding metrics altogether. Some firms, even large ones, say that they don’t need to measure their AR because that function has strong corporate support or because they know what the analysts think. Sometimes thus just reflects the (understandable and real) fear of knowing or of being held accountable for performance. However, it exposes the AR team to a substantial risk, especially if there is a turn-over of management.
  2. Using measurements with a bias towards you. One large vendor, for example, tracks the profile of competitor brands in reports mentioning its brand name. By definition, their method means that every report mentions the vendor itself; as a result they are guaranteed to come first. A more common approach, especially with analyst perception audits, is to not survey a random sample of analysts but instead to select analysts who you consider to be important. This approach means you don’t select analysts who could be equally influential on your clients, or more, but who you have less awareness of because (for example, because you are mot paying them).
  3. –  Collecting overwhelmingly domestic data. I see organisations in North America basing their measurements on samples that are 85% or 90% North American, when that does not reflect the influence of analysts, the universe of analysts they are targetting or their firm’s revenues. If you’re collecting, for example, share of voice data then there’s no reason not to look at German-language (for example) research. You might not speak German, but you can still search to see how often you and your competitors are mentioned. if you see something interesting then get a German-speaking student in and put a pizza in front of the screen.

What are your thoughts?

Duncan Chapple

Duncan Chapple is the preeminent consultant on optimising international analyst relations and the value created by analyst firms. As the head of CCgroup's analyst relations team, Chapple directs programs that increase the value of relationships with industry analysts and sourcing advisors.

There are 3 comments on this post
  1. June 26, 2009, 9:43 am


    Overall I agree with you.
    Just one comment on (1): despite you presenting this as a bit strange -and I agree it may be to some extent until you think about measurement costs.

    If you’re aligned correctly with your constituents and showing the value you deliver -why spend time and money reporting and measuring?

    I do believe many AR managers get trapped into measurement-trying to quantify and report the value they deliver. Some do this well, but many times further questions arrive -dragging the AR manager in a never ending “well or reporting granualrity” to provide more and more details.

  2. June 29, 2009, 3:26 pm

    Hi Ludovic,

    I’ve written a little more about this here: https://www.influencerrelations.com/?p=1246

    Just because an AR team has good internal support, that doesn’t mean it is optimised or that support will continue.
    – Most AR effort is wasted at most firms, because it’s not allocated to the right analysts at the right time. With metrics you stand a chance to see if you could allocate resources better.
    – AR and other communications managers know that when new managers change often follows, regardless of the alignment and value involved. Good metrics help teams to defend and extend their resources
    – Measurement is only a trap if the firm sets targets that either don’t matter or cannot be measured. Otherwise, measures help AR teams to be visibly more effective and win stronger support.

  3. February 26, 2010, 3:44 pm

    […] then I’m really not surprised to hear that – and not only for reasons to do with bias and method. But even if your firm is doing great at AR then there is a lot to learn. For example, Cisco […]