Points on Industry Research

Cat on a phone, calling Extenda

Working in an industry that is not exactly front and center of the tech world means that we get pretty scant mainstream press coverage and have surprisingly little data about our favorite subject, ourselves.  In practice, this means I have to fire up those ol’ Statistics 101 and critical thinking neurons all the time to question industry research. Often this research is dragged into the sales process, used to prove a competitor is the Most Awesome. I want to take a look at just one such report to show how a headline statistic thrown out by a vendor might not be as helpful as they want us to believe.

Nemertes Research provides research for a variety of enterprise software and technology verticals and markets, and is oft-cited by a certain competitor we seem to run into on every deal.  Their 2013 report “Operational Cost Drives Stark Differences in First-Year Telephony, UC Costs” is actually pretty interesting and, on balance, a great overview for an IT or C-level manager thinking about biting the bullet and implementing a phone system from this century, or the underling hoping to build a business case for just such a move.

Where companies have to be careful is when vendors use the findings of an independent consultancy report to buttress arguments made when trying to take your money earn your business. For example, this report shows that NEC and Shoretel have a very low total cost to a business in the first year of implementation. The consultants have thoughtfully calculated not just the actual phones and whatnot you would need off the bat, but first year maintenance, the time your staff spends on the implementation times their pay, and other things typically considered ‘soft costs’ and left off the vendor proposals.

Upon closer inspection, it is important to note that there is a very small sample size to draw from.  I would not fault Nemertes – goodness knows it is hard to get this information out of busy IT people who are probably huddled in a corner in the fetal position after finishing their VoIP implementation. Nonetheless, the survey was based on what appears to be 211 valid responses to uncover information about 7 brands, so about 30 responses per brand or so.

Where I will fault all of these industry consultancies is with the release of their data sets. None of them release their data sets with the report. I find it so odd that companies are allowed to produce statistical claims, and have these claims repeated for real dollars, and there is no peer review of the data behind the claims. I have no doubt the data is valid, but I took enough statistics and econometrics to know that you can also justifiably lop off outliers – as the survey takers state that they did – to change your results, in the name of proper data modeling.  I am not singling out Nemertes at all – I am pointing a finger at the weird tech shadow world of report writers that aren’t regulated and aren’t really held accountable for peer review of their work, for stats that are wildly off, for bad predictions, and bias towards specific brands and solutions.

Of these 30 responses, the Avaya and Cisco responses seem to come from pretty large companies.  The median size of the Avaya sample company was over 1,000 users, and for Cisco, over 500 users.  In my world (moderately successful Mitel/Zultys/NEC dealer in the 2nd biggest city in the US), customers of this size would be my VIPs.  And my VIPs tend to have very robust, sophisticated call centers attached to their phone systems that drive up costs of implementation, or in the case of  this survey, the per seat costs. In fact, both of these companies are very active in the contact center market, much more so than NEC and Shoretel, companies that do not have a robust native contact center option at the time of this writing.

The survey does break down costs further in by brand and for <1000 and >1000 users. That Avaya has start up costs of $1,000 a seat, far ahead of the other brands, in the sub-1000 user market makes me wonder if they did have some contact center apps in these installations. When I compete with Avaya, they are not typically twice as expensive as my products, if we are pitching to the same requirements.

Meanwhile, the winners of the Total Cost of Ownership challenge, NEC and Shoretel, have median company sizes in the survey of… well we don’t know.  We know the NEC range is 6 to 1,500 endpoints (phones), and the Shoretel range is 85 to 1,850 phones.  The largest Cisco system in the survey is 175,000 phones(!), for some contrast.  If we assume a price tag of $500 a user, netting $87.5M, a single Cisco sale did what Shoretel as a company does in a quarter.  My point is that these companies are playing in very different leagues. As an NEC dealer, I can tell you that the 6 phone NEC is dirt cheap, and will not ship with much that an enterprise user would recognize – there is no desktop video conferencing or real-time chat/supervision/reporting baked into the deployment.

There is a lot of great information when you dive into the operational costs, and specifically the information on Microsoft’s Lync solution does match up to what I hear from customers.  Namely, like an Italian car from bygone eras, they are not so expensive to install, but extremely expensive to keep up.  In the case of Microsoft, it is not so much reliability as it is that the mechanics are expensive, though this may come down if Lync really takes off as a viable phone system competitor. The report notes that third party spending on Microsoft is high because the product is new, but I would add it is also because the product is sophisticated and outside the wheelhouse of your more typical IT shop that focuses on SMBs and knows Exchange, AD, Server and the desktop just fine, but that might not have the requisite certifications to even get support on the Lync product.  Microsoft professionals with the alphabet soup of certifications on their business card and the pedigree required of high end IT shops tend to charge a lot more per hour.

Similarly, Cisco’s high SmartNET costs show up in the data, and Avaya, NEC and Shoretel are not terribly expensive to keep up down the road, making me feel that the data here matches up to my extensive though still necessarily anecdotal experience. So maybe I just proved a victim of confirmation bias, and you can chuck this post in the bonfire, too. The takeaways I have for a reader of this post stand; question the headline statistics on these reports, and view these stats as a helpful snapshot of general trends, and not the be all and end all on the topic.