Four steps to successful spoken word audits

Efrem Mallach PhD
Efrem Mallach PhD

“You can’t manage what you can’t measure,” so corporate management always wants to measure analyst relations. It’s easy to count briefings and such, but that misses the point. One should audit results, not effort. The result you want, of course, is a shift in your favor (or favour, for the Brits reading this) in analyst recommendations.

You can get a handle on that by reading what analysts write (reports, newsletters, blog posts and so on) and what they say when quoted in the press. However, what you’d really like to know is what they say to clients on the phone or in person, when nobody is listening and vendor clients won’t hear them. At firms like Gartner, an analyst must use the firm’s official position as a starting point, but can diverge from it to reach an unexpected conclusion. At others, even this much adherence to the “party line” isn’t required if the analyst feels a user’s situation calls for a different approach. How can you become the proverbial “fly on the wall?”

Audits as a compliment to win-loss

There are a couple of ways to find out what analysts say to user clients. One is sales campaign win-loss analysis: debriefing sales reps (often via a short written form) soon after every prospect decision, win or lose. Another is a “spoken word audit,” as suggested by Dave Eckert with Carter Lusher just over nine minutes into their first podcast. This uses your (the vendor
client’s) analyst inquiry time to “play user” with analysts to see how they respond, what they recommend in scenarios you define ahead of time.

A concern about spoken word audits is “Will analysts respond as they would to a user, if they know who’s calling?” An anonymous call isn’t an option, for several reasons. First, you’d need to find a real user who will let you use some of their inquiry time for your market research. Second, it runs the risk of souring the analyst relationship permanently when he or she finds out – as will inevitably happen. Third, it doesn’t let you ask the analyst about more than one situation, that of this user. Hiding behind a real user’s name, then, isn’t a viable option.

The good news is that most analysts get into a certain mindset when they work through user situations. They have no difficulty staying in that mindset when it’s not a real user on the phone. You may want to make sure this is understood in advance, but analysts will be OK with it.

Four steps

IDEAL AR audit
IDEAL AR audit

To do a spoken word audit, you need to do a few things:

  • A. Define the scenarios clearly. A brief description isn’t enough. You need that to give the analyst as the conversation starts, of course, but work it out at least one level deeper to respond to clarification questions without fumbling or making up things that don’t make sense. You may need help from your internal subject matter experts to do this.
  • B. Develop a structured form for recording answers. This is especially important if you contract the audit out. For each competitor, you have to record points in favor, points against, which points matter in this scenario, why they matter, and the overall conclusion. If you’re looking for how well the analyst understands your positioning, independently of who the other vendors might be, put down specific points you’d like the analyst to reflect and what he or she should say about them. You should also note if the analyst brought them up without prompting, mentioned them after a general hint, discussed them in response to a direct question or didn’t know about them at all.
  • C. Do the first audit to develop a baseline. The information is important, to be sure, but what’s really important will be how this information changes over time. If you don’t expect to be able to do a second audit, the value of the first is reduced.
  • D. Repeat the process six to twelve months later with the same firms, whenever possible with the same individual analysts. Less than six months is too short an interval for anything you do to have an impact. More than twelve can let problems go on too long without corrective action.

As a side benefit, even if your measurement-hungry management doesn’t see meaningful results from the first audit, they’ll see that you appreciate the value of measurements. That’s likely to keep them satisfied for a while and to prevent them from coming up with their own, less suitable, ways to measure what you do.

More from Efrem Mallach

AR Classics: Buyer beware on analyst advice

Admit it. You’re an advice junkie. Need proof? The industry analyst business...
Read More