Month: October 2018

New codec on the block – recording failures and CUCM 11.x

The inspiration for this post comes from spending way too much time trying to solve an issue that would have taken only minutes had I been aware of the key information I’m generously bestowing upon my fine readers today.

The problem started with reports that just 8841 phones weren’t successfully recording. I happened to have a spare 8841 phone, so I set up a line, configured recording parameters, and began testing. Sure enough – the recording failed on the spare phone.  I also happened to have a 7942 phone, set up an extension and the recording worked just fine.  Note that I had previously examined the endpoint configurations like recording server profile, Built in Bridge enabled, recording calling search space, and recording server setup for the extension.

Thinking this might be related to a SIP versus SCCP issue, I employed the RTMT to see if the audio for the SIP calls was being forked and sent to the recording server. I was able to drill into the SIP call placed to the recording server, check out the Call Flow Diagram, and confirm the recording server was invited to the party.  If you’ve not done this before, just need to log into the RTMT and navigate to Real Time Monitoring Tool -> Voice/Video -> Session Trace Log View -> Real Time Data. While snooping around all the SIP calls to the recording server, I noticed some successes belonging to 8841 phones.  Intrigued that my problem might not be model dependent, I hopped over to the recording server to see which extensions on 8841 phones *did* record successfully.  Which is where things began to make even less sense – only intermittent failures on some 8841 phones in question, others never recorded at all.

Using my any-excuse-for-a-packet-capture philosophy, I setup two 8841 phone endpoints to allow spanning to PC port and fired up Wireshark.  My test recording failed, but my packet analysis hit pay dirt.  When filtering for RTP, I saw PT=OPUS in the Info column.  Immediately, I had my answer.

I was vaguely aware the Opus codec was a thing, but I previously had no idea that 8841 phones supported the OPUS codec and that CUCM 11.X enabled the OPUS codec automagically (thanks?) – all information I gleaned from this link.

Knowing that recording servers historically hate anything that isn’t g.711 or maybe g.729, I immediately proceeded to follow the instructions from the aforementioned link to find and disable Opus for recorded phones. I previously did this for g.722 many years ago, which is why this solution stung a little, my not being aware there was a new codec on the block and to have preemptively avoided this issue entirely.

from https://www.cisco.com/c/en/us/support/docs/unified-communications/unified-communications-manager-callmanager/200591-OPUS-Codec-Overview.html

While looking at the Opus parameter, I couldn’t help but notice iSAC was new to me as well.  Sifting through my packet captures, I found RTP streams using that codec as well and so disabled it for recorded phones, too.

Below are a couple of screen shots of what you can expect to see in the SIP/SDP packets if you are experiencing this same issue. Hopefully this saves you a bit of leg work should you have some recording failures after an 11.x upgrade.

Feel free to send thanks in the form of flowers, coffee, and cheesecake.

Published 10/18/2018

“Thirteen hundred APs, no open support tickets” – achieving quality in wireless networks

“Thirteen hundred APs, no open support tickets,” Sudheer Matta, VP of Products for Mist Systems, boldly stated during his MFD3 presentation.  At the time, he was referencing one of their largest customers specifically, but the company’s desire to prevent bugs, create high quality customer experiences, and resolve issues quickly were principles that permeated the discussions with Mobility Field Day delegates.

Mist leverages several key components in order to pull off their customer focused reliability, visibility, and proactive troubleshooting of the wireless network.

Cloud-based micro-services architecture.  This modern approach to building systems is part and parcel of what many cloud companies have been doing with their software architecture over the last few years.  Instantiating distributed services and leveraging APIs between these services is foundational to providing the kind of resiliency and redundancy cloud makes possible and Mist credits this architecture with how they are able to push out new features, fixes, and services weekly without causing any data plane outages for customers.

In his presentation, Sudheer shares an impressive case of how Mist was able to do a complete restore for a customer that had deleted their entire controller infrastructure. All the controllers and services were back online in less than 2 hrs with no access point reboots or data plane outages, a feat Sudheer also credits to Mist’s distributed architectural approach.

Analytics. These days collecting data is table stakes, the real advancement is in building better algorithms that provide useful information to customers. Mist calls these, “actionable insights” and they are more than just increasing the noise floor with more alerts. Mist believes their actionable insights are so dead on that they’ve announced proactive anomaly detection, meaning the system will open a ticket on your behalf when an issue is detected.

And the analytics don’t stop with just ticket opening – MARVIS (Mist’s AI) is getting several feature enhancements focused on improving the troubleshooting process, reducing analysis time, and improving RRM.

A culture of attention to detail. After watching Mist’s MDF3 presentations, I would describe their business model as “just good enough is not good enough for us.”

Besides a distributed architecture designed to minimized the number of bugs and the impact of those bugs that do make it into the system, issues are expected to be resolved quickly and not allowed to fester or be ignored.  A clear emphasis is placed on quality and usability of the system, from the architecture to the user experience.

Mist is also listening both to its customer base as well as wireless engineers. An improved adapter bracket, the transparency with firmware version issues, the coming soon red and green buttons, and the constant tuning of the virtual assistant were just a few indicators from the presentations that customer experience and usability not only matter, but are at the top of the priority list.

For more Mist goodness, be sure to check out these posts:

@badger-fi – Mom’s love Mist

@rowelldionicio – Demistifying Wi-Fi Issues

@Drew_CM – Mist Enhances Machine Learning Capabilities To Improve WLAN Performance, Troubleshooting

@theitrebel – MDF Day 1 Recap

Disclaimer: While Mobility Field Day, which is sponsored by the companies that present, was very generous to invite me to the fantastic MDF3 event and I am very grateful for it, my opinions are totally my own, as all redheads are far too stubborn to have it any other way.

Published 10/7/2018