RSS

Voice basics: troubleshooting a failed outbound fax

Faxing is a technology that instead of nuking it from orbit (the only way to be sure), we’ve propped it up and tried to make it part of the VoIP world, resulting in a whole lot of troubleshooting and whole lot of bang-head-here moments for voice engineers.

While time, variances in equipment, and sheer PTSD keep me from exploring all the ways in which faxing can suck go wrong, I thought I’d throw out a recent example of an all too common occurrence – proving your fax machine isn’t the (biggest) offender in an outbound communication failure.

Specifically, this example deals with an XMedius fax server, a Cisco voice gateway with PRI, and a who-knows-what fax endpoint on the other side.  Your mileage in fax troubleshooting may and likely will vary, just keep that in mind and a drink at hand.

The first step in dealing with one of these reported issues (after cursing, of course) is to determine if it’s an isolated incident or possibly a dialing issue.  Besides calling and confirming* a fax machine actually picks up, checking your inbound and outbound logs on the fax server can quickly quell those reports the server is down when someone really forgot to dial a 9 when sending the fax. Happens all the time.

In my case, I had plenty of inbound/outbound successes to determine this was an isolated case.  I also had the packet capture feature of XMedius turned on.**

This feature is brilliant, truly not an understatement.

I opened the packet capture for one of the failed attempts, navigated to Telephony -> VoIP Calls -> and then selected Flow for my call.  When you do this, there will be quite a bit of information presented in graph form.

You should be looking for a few basic things in particular:

  • Do you see the call ever connect?
  • Do you see the sender’s cng (calling tone) packet?
  • Do you see a DIS (Digital Identification Signal) from the remote endpoint?
  • Do you see the sender’s training message?
  • Do you see the remote endpoint’s CFR (confirmation to receive)?

In my flow graph of the not-so-happy fax, I notice that even though I’ve made contact with the (whiny) fax machine on the other side and negotiations have been successful – the remote endpoint never sends a CFR, therefore the server will not send the fax data.

The fax server tries again and again to elicit a response, but there’s only silence from the other side.  I assume because the remote endpoint realized that for every successful fax, a puppy dies.  Well, that’s the rumor I’ve heard (or started).

Here’s an excerpt from the flow graph, definite lack of CFR.

No CFR

Below is flow graph of a fax that the server sent successfully to another number.  While there are differences, you can see that CFR goodness the flow graph above is missing.

Successful Fax

After reviewing this information, I moved onto finding out if the voice gateway ever sees the CFR and maybe just forgets to send it along.

After working with TAC and doing a PCM capture on the gateway, I was able to confirm that the remote endpoint never sends the CFR, which meant I could declare with some amount of relative certainty that this was a whole lot of not-my-problem.***

TAC even provided me this handy-dandy flow graph built from the captures we took on the gateway, you can see that the fax server tries three times (TCF (9600)) to get the remote end to cough up a CFR, but no dice.

outbound fax flow

While this just scratches the surface, these basics, along with a formidable hammer, should get you started in your fax fighting mission. Just remember to really effectively troubleshoot a fax machine, it’s all in the swing…

 

Published 4/10/2015

*Do not skip this step. Never assume a user is asking you about problems with a working telephone number.  Always test from outside your phone system to confirm that the phone number in question hasn’t been disconnected or written down wrong by the user. This will save you countless hours and possibly what’s left of your sanity.

**Check your XMedius Administrator’s guide or call their support for steps to turn on this feature, it’s a pretty straightforward process and well worth the time.

 ***Trust me there are no absolutes in fax, unless you’re talking about frustration, that part is guaranteed.

 

Tags: , , , , , , , , ,

Short and sweet – how to block an incoming call on your voice gateway

Welcome to a quick post on how to block an incoming call when you know the calling number you want to block. Specifically, this is how I would block an incoming call on a Cisco voice gateway with an ISDN PRI attached. Your mileage might vary a little with SIP trunks and will definitely vary quite a bit with MGCP.*

The first thing you need to do is create yourself a voice translation rule, something like this ought to do the trick:

voice translation-rule 9
rule 1 reject /5550005555/   <<keep in mind this is the calling number you want to block, but I like to test initially with an outside number such as my cell phone that I can test with.

Then set yourself up a lovely translation profile that references the rule you just created. Name it something obvious so that the next administrator doesn’t have to beat you to death for your obscurity:

voice translation-profile CALLBLOCK
translate calling 9

To complete the configuration, add these two commands to your incoming POTS dial-peer.  If you aren’t sure what your incoming dial-peer is, use the debug voip dialpeer all command and make a test call.  This is a good idea even if you think you know what the inbound dial-peer is because sometimes life is whimsical, and dial-peer configurations even more so.

dial-peer voice 4445 pots
call-block translation-profile incoming CALLBLOCK
call-block disconnect-cause incoming unassigned-number

There are a few ways to test this.  As I mentioned before, you can use your own cell phone number in the original configuration and confirm that the call blocking works. Then just substitute the to-be-blocked number into the voice translation rule.

You can also run the following command and see what the router *thinks* it will do when it sees the number you are trying to block:

test voice translation-rule 9 /5550005555/
/5550005555/ blocked on rule 1

As with all things voice, there are eleventy-billion ways to accomplish a task, this post just covers one.  If you have another method you prefer, please share in the comments, would love to hear it.

Published 03/10/2015

*The process with SIP trunks is practically the same, your inbound dial-peer won’t be POTS, though.  MGCP will require you to use CUCM 8.0 or later for this, check out this document

 

 

Tags: , , , , , ,

Changing your Unity Connection SMTP domain

Changing the SMTP domain on a Unity Connection server really isn’t that big of a deal, but as with all things voice, no change works as initially advertised.  Previously, I had never had cause to mess with the SMTP domain address, but recently one of the major cell providers quit delivering our voice mail message notifications to devices and, not surprisingly, users were none to happy about it.

My guess was that the carrier in question didn’t much like the format of the sender address, since it included a sub domain: unityconnection@myservername.mydomainname.com.  I quickly decided that changing the SMTP domain on the server would easily test that theory and seemed far less painful than opening a ticket with a large service provider*. I did open a TAC case just to see if there were any caveats in making this change I might want to be aware of. That’s a whole lot of voice experience talking…err, writing. The voice paranoia runs deep for a reason.

TAC indicated that not only was this a simple change as thought, but that only one service would have to be restarted – the Connection Conversation Manager, and that wasn’t to be a big deal. Well, finding that hard to believe**, I proceeded to make said change and found that there’s a little more to the story.

First – there are *three* services that have to be restarted, and since two of them are critical services, you experience a failover if you are running in HA. The system does warn you this will happen, and for what it’s worth I did not experience a loss of service doing this. Certainly don’t blame me, though, if you do have an outage and aren’t in a maintenance period when you attempt this change.

smtpUnity

 

Of course, you get to rinse and repeat on the secondary server in an HA environment. Personally, nothing about the warning prompt I got on the secondary server indicates this is not a big deal, but hey, maybe that’s just me…

smtp-connection conversation manager secondary

That being said, once the services were restarted, and my blood pressure returned to normal, I expected to see the SMTP domain updated and a happy dance to ensue.  Alas, that was not the case.  After consulting with TAC, and without the least bit of surprise whatsoever, I found that a reboot was “sometimes”, infer always, required.

This did fix my problem and the delayed happy dance was epic. And thankfully not recorded for the sake of my remaining pride.

 

Published 1/26/2015

*Almost all levels of hell are more pleasant than opening a case with a carrier. I suspect Satan actually admires the ingenuity of carriers and any future levels of hell are modeled on their expertise and innovation in human suffering.

**There is seemingly far more proof of the existence of the Loch Ness Monster, Big Foot, and the Abominable Snowman than of something involving voice being “easy”

 
2 Comments

Posted by on 2015/01/26 in Unity Connection

 

Tags: , , , ,

Runt Post: HP Discover Notes

Last week I had the privilege of attending HP Discover in Barcelona and thought I’d hit the highlights while they were still fresh in my mind.

HP’s continued OpenFlow work stood out last week as HP has several applications leveraging OpenFlow available to customers now. One of the things that I find most appealing is that there are actually *campus* SDN applications, not just data center applications. Recently, data center has received all the love in networking innovation, leaving campus networks to the same old same old. Campus networks, however, represent a wide range of potential for SDN, so it’s nice to see some OpenFlow applications focused in that direction. The Network Optimizer application that dynamically allocates bandwidth for Lync experiences and the Network Protector app that leverages a TippingPoint Reputation database are the HP SDN applications I’ve heard the most about, but a look at the SDN App store shows there are quite a few others out there, customer ready and available.

HP’s Intelligent Management Center caught my attention when listening to the Packet Pusher’s episode on the platform and after talking to Chris Young about the product at Discover, I am all the more curious to get a demo up and running.  IMC not only allows for your basic network management and monitoring tasks, but also offers advanced features such as config validation and device configuration from a centralized management console. It won’t get rid of all your other single pains – err – panes of glass, but does look quite promising for centralizing network management in a way that doesn’t suck your will to live as some of the larger, more bloated platforms tend to do. Also, support of third party devices is big for anyone not running HP gear exclusively or even at all. The insight into ESXi servers also caught my interest as being super cool – a way to see into what those wily sys admins have done with their virtual switches while they blame your physical switches for the problem.

I also found the work HP Labs is doing to be quite fascinating.  Having an increased R&D budget as of late, the HP lab geeks are taking on some pretty cool projects. Much of their energy is being funneled into photonics and memristor technology projects, collectively referred to The Machine. Personally, the name “The Machine” sounds a bit over the top, but there is some serious science going on in this line of research and my geek DNA can’t wait to see what develops from these endeavors.

I had a highly enjoyable experience overall and loved getting to geek out over tech with some other seriously fabulous nerds – you should check them out as well because they are *awesome*.

Published: 12/9/2014

 

Disclaimer: While HP Networking was very generous to invite me to this fantastic event and pay my expenses, and I am very grateful for it, my opinions are totally my own, as all redheads are far too stubborn to have it any other way.

 

 

Tags: , , , , , , ,

Runt Post: Unity Connection 8.6 and SU5, there’s a .cop file for that

I ran into a fun* issue last week while upgrading Unity Connection from 8.6.(2a) to the latest patch SU5, and I would tell you to be sure to read the release notes on the subject, but as I found when calling TAC at 2am, the release notes don’t warn you about this particular issue.  See, after the upgrade, all those precious call handlers you painstakingly configured and tested over the past few years just don’t work. At all. Not even a little. In fact, the system just routes these calls to the Opening Greeting. Much to your frustration, you can see that the extensions are still defined on the call handlers just like before the upgrade, but after the patch the system rudely snubs your call handler configuration entirely.

As much as I am totally ruining the surprise of your discovering and experiencing this precious jewel of an error on your own and wondering what the heck to do about it, there is a .cop file that fixes the issue: ciscocm.cuc_86x_cdl3.cop.sgn (cco login required).   After uploading the file to both servers (assuming an HA environment), a reboot of the primary and then the secondary is all that’s left to do.

And people wonder why voice engineers drink…

 

Published 11/03/2014

*voice engineers have a slightly skewed definition of fun, precisely because of issues like this…

Note: there is a bug ID for this, CSCuq63776 (cco login also required) in case you are interested.  I have no idea if every upgrade that goes from 8.6(2a) will hit this particular bug, but at least now you won’t be surprised if you do…

 
 

Tags: , ,

The 8945 firmware upgrade dance

For awhile now I’ve been hitting the fabulous issue mentioned in this Cisco forum post for 8945 phones in which the phone basically stops incrementing the time display. Being as I only had a few of these deployed and unplugging/plugging them would fix the issue, I had put off upgrading them. I had plan to put off upgrading the firmware until the proverbial upgrade cows came home and sat around my desk mooing their demands for attention, but unfortunately my plans were udderly derailed*.

I discovered while reading various release notes that the usually cumbersome but predictable upgrade process would be a bit more involved and would need to be done before I started major version upgrades, otherwise my phones would be shiny dialtone-less bricks with no usable firmware to make their happy little phone lives better.

Cutting to the chase, here’s what you have to do if you happen to be in the same series of release boats that I’m paddling around in.  Keep in mind that in this particular situation I am starting with 8945 phones with SCCP894x.9-2-3-5 installed and CUCM version 8.6.2.20000-2, your mileage may vary:

  • You must have the minimum Device Package 8.6.2(22030-1), which in this case I didn’t have. 8941 and 8945 phones won’t register at all if you try to install 9.3(1) or later without this device pack installed first. Yay.
  • Next, you’ve gotta do a step upgrade from 9.3(4) to 9.4(x) because in the grand tradition of voice upgrades being about as much fun as root canals, you can’t just upgrade straight from a 9.2 release to a 9.4 release. Double Yay.

There are a few key things to keep in mind when doing device pack and firmware installations.  First and foremost, always always always read the release notes. If something doesn’t seem clear or make sense, open a TAC case to clarify. This is way better than blowing up your server or phones because you assumed something the release notes didn’t cover or explain properly.

It should go without saying that you should always confirm you have a good backup before doing any file installations at all. I’m saying it anyway, check your backups, never assume they ran. Trust but verify, the emphasis being on verify.

Remember to copy down the values on the Device Defaults page before the upload of the files and to paste in old values if necessary directly after the upload. Check out this previous post for why you will thank me for this later.

Lastly, always remember to reboot after device package installs and to stop/start the TFTP service after firmware file installs.  These processes will save you some heartache and potentially a bruised skull from repeated head-desks when you finally realize you never did stop and start that service and the last 45 minutes of troubleshooting was for nothing.

*Anyone with a beef about my cow puns probably shouldn’t follow me on twitter either, the puns only get worse more fabulous from here… 

Published: 10/23/2014

 

Tags: , , , , , ,

Getting to know Cisco ACI…

Watching Cisco present on ACI at Networking Field Day 8 was a nice expansion to the introduction I received on the product almost a year ago at the ACI launch event in New York.  Now that APIC is shipping, companies can swap over from NX-OS mode to ACI mode and start playing with the magic that is application network profiles.

The basic components of ACI are the fashionable spine/leaf architecture that is all the rage these days, an APIC controller talking OpFlex southbound, and a switch operating system on Nexus 9Ks that interprets policy and forces it down to the endpoints.  Underneath the covers, each switch uses ISIS to build a routed topology from any VTEP (virtual tunnel endpoint) to any VTEP (basically from any leaf to any leaf). Rather than the controller programming routes and handling traffic forwarding, the controller focuses on pushing down policies that are understood and then implemented by the switches.

The concept of a self provisioning network also comes into play with the ACI solution, as does the concept of one big fabric to rule them all. The fabric can be zero touch provisioned, with the controller finding new switches brought online. The controller also acts as a single point of policy provisioning – the fabric itself scaling up to 12 spines, with multiple active controllers all sharing data for redundancy.

The heart of ACI really lies with the policy model and associated concepts.  ACI works by putting things into groups – usually done with identifiers like vlan/vxlan ID, subnet, 802.1q tag, or by physical/virtual port – and then these groups are assigned policy contracts which basically “turn on” connectivity between these groups, according to the rules of the policy assigned. The level of abstraction inherent in these contracts lend themselves well to automation and consistency in network policies, as well as allowing for a clean up process as applications are removed – therefore solving some of the what-the-heck-was-this-thing-nobody-remembers problems we engineers often encounter.

Application network profiles can be home brewed as well as provided in the form of device packages from vendors.  These device packages will automagically roll out the best practices for the application at hand, and if they are from an official partner, TAC will even handle support issues that arise from using the package. As Joe Onisick put it, “think of it as an automatically deployed Cisco validated design.”

There’s much more covered in the Networking Field Day 8 presentations, including service graphs – think service chaining but with flexibility for differing behaviors for various traffic groups, an API inspector that allows you to see the API code as you make calls through the GUI so you can create automation scripts from it, and atomic counters which allow for detailed health scores and packet tracking, but as I’m a sucker for a good demo, I’ll leave you with this, Paul Lesiak showing off APIC’s mad programmability skillz.

 

Published: 10/14/2014

For more links to ACI resources, you can check out my previous post, check out some excellent videos by both Lilian Quan and Joe Onisick on the subject (just go to youtube and seach for Cisco ACI), or check out Lauren Malhoit’s blog where there’s some good posts on getting to know ACI as welll.

Also, mucho bonus points to Lauren for not only being generally awesome all the time, but for also providing this ginger with a desperately needed Diet Coke as a caffeine source at this 8am presentation, an un-caffeinated ginger is a scary, scary thing.

Disclaimer: While Networking Field Day, which is sponsored by the companies that present, was very generous to invite me to this fantastic event and I am very grateful for it, my opinions are totally my own, as all redheads are far too stubborn to have it any other way.

 

 
1 Comment

Posted by on 2014/10/14 in ACI

 

Tags: , , , , , , , , , , , ,

 
Follow

Get every new post delivered to your Inbox.

Join 285 other followers