Where’s my RTP?

As many of my readers already know, I recently shifted gears from a being a consultant focused on voice to a being network engineer focused on…well, everything. One of the first items on my list to tackle, however, put my years of voice training to good use.

A common issue voice engineers are faced with is one way or no way audio.  As a voice consultant, those types of issues almost always ended with my saying the line “that’s a network issue, you’ll need to follow up with *those* guys” because almost always one way or no way audio is a routing issue of some sort. Of course, now that I am again one of *those* guys that gets to fix the routing issue too, that line is not quite as charming.

Many people who do not do voice everyday are not aware of the fact that the actual audio payload of a phone call is RTP traffic sent directly between the two endpoints.  This is starkly different from the call signalling, which is typically between the endpoint and the Unified Communications Manager. Given that signalling and RTP can take differing paths, you can end up with phone calls that seem like they should work but don’t, and for better or worse, the other end cannot hear you screaming in frustration.

My most recent run in with this issue involved a new VPN connection setup and users unable to hear voicemail prompts when checking messages. Complicating the issue was the fact that “normal” phone calls – phone calls between the remote and main site, were working just fine. A closer look, however, easily explained why one was working and not the other.

A good place to start on a no audio issue like this is to make use of the Question Mark button located on most Cisco phones. In this case I was dealing with a 7960 (don’t judge me) and just clicking the Question Mark button twice pulled up stats for the conversation.  Most notably, I could see that the transmit packets were incrementing, but the receive packets were not*. Newer phone models like the 69/8900 series hide this information in some other menu which isn’t entirely obvious, but just check the documentation to see how to pull up phone call stats. You can see in the photo below the type of useful information you can get – note the Question Mark key is in the bottom right of the picture.

phonephoto

Having confirmed that the phone thought it was sending RTP packets, but clearly not receiving them, I decided it was a good time for a packet capture**. One of the things I love about Wireshark is how easy tracking down RTP conversations is in a .pcap file. To take the capture, simply plug into data port on the phone, launch Wireshark, start the capture, and make your test call. Then stop the capture, click on Telephony -> RTP -> Display All Streams. You will see the RTP conversations in the capture, choose the one you want, and then click Prepare a Filter to view just the RTP packets you are looking for. Be sure to wave your hand and mumble “these are the RTP packets you are looking for” while you do.

Once I had my filter applied I could see the two IP addresses that the RTP was attempting to flow between, one being the phone ip address and the other being the ip address of a secondary voicemail server, not located at the main site that the remote phones were typically calling.  I then tried to telnet to that voicemail server on port 5000 to monitor the voicemail port traffic but my connection was unsuccessful, reiterating that I had a routing issue somewhere.***

Capture

Now the conclusion to this episode is not nearly as exciting as the troubleshooting process, adding a route to the VPN configuration to the correct subnet was all that was required, but it serves as a great example of when a voice issue isn’t actually a voice issue, no matter how much it looks like one.

Published 5/25/2013

*Another item to rule out on these types of remote site setups is the possibility of a codec mismatch.  The default between sites is typically g729 and not all voicemail servers can tolerate that.  Using the Question Mark key can show you the codec that is being used and then you can investigate whether or not the proper transcoder resources are in place.

**Pretty much anytime is a good time for a packet capture.

***I did a write up on how you can monitor voice ports with Unity Connection 8.6 using telnet here

Just want to be heard…

One of the things call centers supervisors really like to do is listen in on agent calls.  I’m sure it’s not *just* because they are nosy-type people, [insert business justification here], so part of my job is to make sure their eavesdropping is configured and working properly in Cisco Contact Center Express (UCCX).

Now there are about 11 ways to Sunday monitoring and recording can be jacked up by various elements, not the least of these being the voice engineer at the configuration helm. So when, during a deployment, it was found that calls were not able to be monitored or recorded, I skipped right past the look of surprise and moved straight into the what-is-it-this-time expression.

First, the symptoms.  Agents were getting calls and their supervisors were recording these calls. This means a whole bunch of agent/supervisor/phone setup tasks were completed correctly. Plus one point to the competent voice engineer with the mad skills. The recorded files were then being played back, however, and the tracks contained no audio.  Minus one point to the slightly less competent voice engineer who may, in fact, just be mad.

This not being my first rodeo, I initially suspect a codec issue, quickly confirmed by using the question mark button on the phones when the calls are made.  The display on the phone shows me the codec the calls are using is g.722 which, while a lovely codec, is not actually supported by UCCX.  It having been a long day, I decide to take a hatchet to g.722 and disable it in the Call Manager system wide parameters – ensuring no more g.722 EVER. Or at least not in this cluster.

Fully expecting the new rounds of tests to be successful, I get to use my surprised look after all when, once again, the recorded tracks lack audio. Grrr.

Firing up trusty Wireshark shows something very interesting – there is no RTP traffic from the PC to the UCCX server. For those who don’t eat, sleep, breath voice, RTP is the transport for the audio portion of the call.  All the setup/control messages will generally use SCCP, SIP, or H323, but the packetized voice uses RTP over UDP. The fact that it is completely absent from my capture file is more than a bit disconcerting.

After a nice talk with my buddies at TAC, they inform me that this is commonly seen with the particular brand of antivirus being run on the client workstations. After uninstalling the antivirus product and running the capture file again, RTP packets make an appearance and victory is declared in my favor.  100 points to the cheeky voice engineer from Dallas.

In case you were wondering what this RTP traffic looks like in Wireshark, you are looking for something like this:

RTP Traffic

As an added bonus for making it to the end of this post, here are a few other things you should check on the phone device configuration page in Call Manager when having issues with recording and monitoring, they are pulled from this document: Cisco CAD Troubleshooting Guide CAD 8.0 for Cisco Unified Contact Center Express Release 8.0 Cisco Unified Communications Manager Edition revised April 2011

  • PC Port—Enabled. If the PC Port is not enabled, the agent PC that is connected to the port will not have network access. No voice streams will be seen by the desktop monitor module
  • PC Voice VLAN Access—Enabled. If the PC Voice VLAN Access is not enabled, no voice streams will be seen by the desktop if the desktop is not a member of the same VLAN as the phone.
  • Span to PC Port—Enabled. If the Span to PC Port is not enabled, the voice streams seen by the phone will not be seen by the desktop monitor module.