New codec on the block – recording failures and CUCM 11.x

The inspiration for this post comes from spending way too much time trying to solve an issue that would have taken only minutes had I been aware of the key information I’m generously bestowing upon my fine readers today.

The problem started with reports that just 8841 phones weren’t successfully recording. I happened to have a spare 8841 phone, so I set up a line, configured recording parameters, and began testing. Sure enough – the recording failed on the spare phone. I also happened to have a 7942 phone, set up an extension and the recording worked just fine. Note that I had previously examined the endpoint configurations like recording server profile, Built in Bridge enabled, recording calling search space, and recording server setup for the extension.

Thinking this might be related to a SIP versus SCCP issue, I employed the RTMT to see if the audio for the SIP calls was being forked and sent to the recording server. I was able to drill into the SIP call placed to the recording server, check out the Call Flow Diagram, and confirm the recording server was invited to the party. If you’ve not done this before, just need to log into the RTMT and navigate to Real Time Monitoring Tool -> Voice/Video -> Session Trace Log View -> Real Time Data. While snooping around all the SIP calls to the recording server, I noticed some successes belonging to 8841 phones. Intrigued that my problem might not be model dependent, I hopped over to the recording server to see which extensions on 8841 phones *did* record successfully. Which is where things began to make even less sense – only intermittent failures on some 8841 phones in question, others never recorded at all.

Using my any-excuse-for-a-packet-capture philosophy, I setup two 8841 phone endpoints to allow spanning to PC port and fired up Wireshark. My test recording failed, but my packet analysis hit pay dirt. When filtering for RTP, I saw PT=OPUS in the Info column. Immediately, I had my answer.

I was vaguely aware the Opus codec was a thing, but I previously had no idea that 8841 phones supported the OPUS codec and that CUCM 11.X enabled the OPUS codec automagically (thanks?) – all information I gleaned from this link.

Knowing that recording servers historically hate anything that isn’t g.711 or maybe g.729, I immediately proceeded to follow the instructions from the aforementioned link to find and disable Opus for recorded phones. I previously did this for g.722 many years ago, which is why this solution stung a little, my not being aware there was a new codec on the block and to have preemptively avoided this issue entirely.

from https://www.cisco.com/c/en/us/support/docs/unified-communications/unified-communications-manager-callmanager/200591-OPUS-Codec-Overview.html

While looking at the Opus parameter, I couldn’t help but notice iSAC was new to me as well. Sifting through my packet captures, I found RTP streams using that codec as well and so disabled it for recorded phones, too.

Below are a couple of screen shots of what you can expect to see in the SIP/SDP packets if you are experiencing this same issue. Hopefully this saves you a bit of leg work should you have some recording failures after an 11.x upgrade.

Feel free to send thanks in the form of flowers, coffee, and cheesecake.

Published 10/18/2018

Voice basics: upgrading firmware for specific phone models

Occasionally in voice world you are required to update firmware for just certain phone types and not others. Say for example you’ve received a batch of those 7945 or 7965 phones that only support 9.3(1)SR1* and you aren’t running that version of firmware on your CUCM yet. There are just a few simple steps and gotchas to watch out for to get this firmware up and running on these phones.

First, hop on over to cisco.com and download the firmware files for CUCM. In my case, I downloaded cmterm-7945_7965-sccp.9-3-1SR4-1.cop.sgn.**

Save the file to your sFTP directory and launch your sFTP client. I typically use FreeFTPd, but use whatever sFTP client you are comfortable with.

Now before you go off installing firmware files, there is one crucial step I would always recommend:

Log into CUCM, navigate to Device Settings -> Device Defaults -> [models of phones you are upgrading]

Copy and paste the current value of the default firmware into Notepad or Evernote or whatever program you use that helps keep you from losing stuff.

In this case, I copied SCCP45.9-2-1S which was the default device firmware for the 7945 and 7965 phones.

Next jump on over to the publisher and navigate to OS Administration -> Software Updates -> Install and Upgrade. Walk through the prompts to upgrade from sFTP, select your file, check the MD5 hash, and start the install.

Once installed, repeat the process on the subscribers as well.***

Now at this point, newbies to voice might think that their work here is done, firmware file installation reported complete, so what could be left? Well, it’s voice, it’s *never* that easy.

In this case, the system has actually done you a favor. If you jump over to CUCM Administration, you will see that the device default fields for your phone models have been updated to the version you have just installed. Because, however, you have not stopped and started the TFTP service, you have a fabulous opportunity to paste back in your old version value if you would like, before the TFTP service is reset and the system starts upgrading your phones. I typically copy and paste the new value to Notepad and place the old value back into the appropriate fields. Now I can apply the firmware to a single test phone and not *all* the phones of that model type before rolling the firmware out. This becomes massively more important when dealing with some of the newer phones and brand new releases of code.

Then it’s over to Cisco Unified Serviceability -> Control Center – Feature Services to restart the TFTP service on each node.

Now, to test your new firmware on a specific phone, find that phone in CUCM and paste the new firmware value into the Phone Load Name field. Reboot the phone and it should then grab the new version.

Once you’ve established that the new code doesn’t bork your shiny Cisco device, paste the new firmware value into the appropriate Device Defaults field for your phone models. Then all phones of that type will be able to upgrade to that version.

*There are certain 7945s and 7965s shipping now that only support 9.3(1)SR1. I missed any official announcement but remembering hearing rumors of this at Cisco Live. The phone will register without you upgrading the firmware files on CUCM, but if you ever reset it to factory defaults, it won’t register again. At that point, I offer you this help on fixing bricked phones

**It should go without saying, but I’ll say it anyway, always read the release notes and check the compatibility of the file(s) you are selecting. Voice is extremely unforgiving if you choose not to read the fine print.

***Technically you really only need to do this on the server(s) that are acting as TFTP servers, but I like the consistency of all the servers having the files in case one should need to take over that role unexpectedly.

Published 3/4/2014

It’s been mostly dead all day…

Setting a misbehaving phone back to factory defaults is a great way to cure endpoint wonkiness*. This process successfully eradicates a multitude of demons, but recently my reset results were more of a botched lobotomy than a successful exorcism. The phone powered on, cycled to the Cisco logo screen, spat out curses about my mother, and power cycled itself again. Cue infinite loop. Please note, it might have been me doing the actual cursing…

So what do you do with a rather expensive Cisco brick besides throw it at well deserving users?

Well, you’ll need a couple of things to pull off this miracle – first, a really large cloak**…nah, only kidding, but you will need firmware files. Yep, head on over to cisco.com and locate the firmware files for your phone model. You are looking for the individual files versus the .cop file for CUCM. In my case, I had a beautiful 7945 brick, so I downloaded cmterm-7945_7965-sccp.9-3-1SR2-1.zip.

You will also need tftpd32 or tftpd64. I guess technically you could use any tftp server application, but this one is free, easy to configure, and I know it works.

All you need now is to build yourself a little network island using a stand alone switch and some patch cables. You will need to provide a DHCP address to your phone with the option 150 set to the IP of your laptop. I used a small layer 3 switch that I could configure a DHCP pool on.

Next up, configure tftpd32 as a server and place the firmware files in the correct directory.*** Rather than reinvent the wheel, here is a good post on configuring tftp32 as a server: http://www.tricksguide.com/how-to-setup-a-tftp-server-tftpd32-windows.html

Once you have your laptop and the phone in the same vlan with the TFTP server running, you should start to see the magic happen. It should look something like this when your firmware files are being downloaded by the phone:

This should bring your phone back to the land of the living, if you’re lucky. But since you are already at this point I am highly doubting your luck, so here is a post with a few more ideas that may help as well: https://supportforums.cisco.com/thread/2007970

Published 12/5/2013

*wonkiness: the term a voice engineer uses to explain the unexplainable behavior of a device that is configured correctly but is clearly possessed by an evil spirit with an extremely vindictive sense of humor.

**Not surprisingly, another Princess Bride reference, just go watch it already.

***Should you forget or not realize what directory your tftp server is set to serve up files from, and yes sadly I am speaking from experience on this one, you can buy yourself a clue in the form of a Wireshark capture. You should see the file being requested in the capture. If you are seeing a request like the one below, then check the location of the file. If not, check your DHCP bindings and confirm the phone is getting an address with the option 150 set.