Expert Q&A: Validating Information Sources When Evaluating Analytics Solutions (Part 2)

This is Part 2 of our expert Q&A with Jen Underwood, Founder of Impact Analytix. In Part 1, we talked about the importance of critical thinking when evaluating business intelligence (BI) and analytics solutions. In this post, Jen digs deeper into the different types of information sources out there and how buyers can determine their validity and accuracy.

What information sources do you find the most useful in evaluating analytics solutions?

Clients are my favorite sources because they will tell you the real deal—if they’ve had issues, if they failed, what they wished they would’ve done. When possible, ask to speak to client references. If they’re not willing to give you references, that’s a warning sign.

Sales reps can also be excellent resources, but keep in mind that they are inherently biased because they make money by influencing you. Most reps are measured by sales quotas, net new accounts, take share deals, and customer satisfaction. To get around this bias, find supporting data, speak to references, and look out for omissions or outdated information.

Research firms are another valuable resource, just as long as you understand the type of research you’re looking at—subscriber paid or vendor paid, media driven, or crowdsourced. Remember that nearly all reports have some informational value. To find it, ask yourself: What was the methodology? How was the information gathered, and does it accurately reflect the larger population? And most importantly, did the evaluators conduct hands-on tests?

Is crowdsourced information valid?

Yes, it can be. But keep in mind that premium research firms may filter the answers—and in these cases, you won’t necessarily find anything negative. To determine whether the information is biased, think about:

• Who filled out the answers?
• How are answers verified and validated?
• Are the answers filtered?
• Were vendors involved in the collection process?
• What incentives were provided?

What about industry awards? There are so many out there today—how do you know which ones are credible?

Some of these are legitimate and some are completely bogus. To separate the good from the bad, think about how the awards are chosen. Did everyone win? Was there an award fee? Is the award merely an advertisement? A real award will share the methodology that was used. If you cannot find that basic information, it’s probably a paid award.

What are the common mistakes people make in evaluating vendors?

Relying on a single report—even one as credible as the Gartner Critical Capabilities for Analytics and BI Platforms—is one of the biggest mistakes you can make. One analyst report only shows you a slice of the embedded analytics market. I recommend reading at least three to four reports from different types of analysts.

Another mistake is under-valuing your own perspective. Every project has its own unique business intelligence requirements. An analyst’s job is to focus on where they think a market is headed, but that rarely matches perfectly with your individual needs.

And finally, skipping the evaluation process is a huge mistake. Spend time getting to know your top vendor and validating their solution against your requirements. Analyst reports may help validate your decision to executives, but the only way your team will know if the solution will work is by digging in yourself.

Any smaller companies you like to follow for specific points of view?

One of my favorite analysts is Boris Evelson with Forrester. He doesn’t just listen to the vendor, he also shows up at conferences to talk to users directly. Another analyst I enjoy is Dion Hinchcliffe with Constellation Research. He hand-tests the solutions himself and shares his experiences.

Learn more from Jen Underwood in the full on-demand webinar >