Upgrading Unity Connection 9.1.2SU2 to 11.5.1SU2

Prologue: This post is intended as an informative prep document for those planning to upgrade Unity Connection (in an HA setup) from 9.1.2SU2 to 11.5.1SU2 using the OS platform upgrade and not PCD (because reasons…) – your upgrade mileage may vary and likely will. These are notes from my upgrade experience and while many of these steps are universal to the upgrade process, each upgrade is a unique snowflake. Do not use this post as a step by step How to Guide. You will be sorry, and I promise TAC won’t be impressed when you tell them that you followed a process from a blog, even one as awesome as mine.

So let’s get started on this upgrade for Unity Connection 9.1.2SU2 to 11.5.1SU2 – the best document to begin with is the Unity Connection upgrade guide found here. Your best bet is to read it through several times and to not be surprised when you can quote any particular paragraph and page number in your sleep.

Also, reviewing the release notes found here is a must.

Once you’ve read the docs and are starting to put together a plan, here’s some guidance on what that plan should entail:

Licensing. You will need some.  Be sure to order your upgrade through PUT and be sure that your licensing server can handle the version of Unity Connection you are going to.  You may need to upgrade your license server or add license definitions to your licensing server pull this off.

Installing a .cop file. Installing a .cop file for RSAv3 keys is listed in the upgrade guide. Digging deeper into this, if you are at version 9.1.2SU2 (AKA 9.1.2.12901-3), this build contains the same fix as the .cop file.

To quote the release notes for this .cop file:

If you are already running CUCM release 10.x or higher you already have this fix and do not need to install this Cisco Options Package (COP) file. If you are running an
Engineering Special (ES)or Service Update (SU)that already contains CSCua88701you do not need to install this COP file. CSCua88701 is included in: 8.5.1.17123-1 and higher, 8.6.2.24122-1 and higher, 9.1.2.11018-1 and higher.

I skipped the installation of this .cop file per this information and also receiving no errors for the .cop results of the “run cuc preupgrade test” output run before the upgrade. For more details, check out this document.

Confirm ESXi support. You will want to be sure the ESXi version your virtual machine is on is supported for the version of Unity you want to be on, and that information lives here.

Determine OVA changes. This step requires checking here and comparing these requirements to your current virtual machine.You will likely need to change your virtual machine vNIC to VMXNET 3 as VMXNET3 is required. You may need to engage a little VMware PowerCLI magic to pull this off.*  Note, if you need more RAM or hard drive space, TAC says a rebuild of the server is required. I’ve heard rumors of engineers changing the memory but the official stance of support is not to touch that stuffs.

Determine your disaster rebuild strategy(s). I recommend downloading and installing COBRAS, getting it running, and exporting all your data from your voicemail servers. With this data, you can import into a fresh build if it comes to that. You should also be sure to have a good DRS backup, which when restored will have all your settings versus COBRAS which will require you to recreate some settings/features.

For me, the basic flow of the upgrade went something like:

Exported data using COBRAS
Confirmed available space on the common partition, need at least 25 GB – I did not have to change high and low water marks, refer to the guide if you need to.
Confirmed nightly backup
Verified voicemail ports registered pre-upgrade
Confirmed HA and replication status pre-upgrade
Ran “run cuc preupgrade test”
Chose not to use utils iothrottle disable command since 24/7 production environment – if I had used this command, after entering it, a reboot is required, re-enabling it post upgrade would also be another reboot
Installed upgrade to publisher, automatic switch version required
Waited for services to establish
Installed upgrade to subscriber, automatic switch version required
Waited for services to reestablish and replication to be good
Powered off pub, ran VMware PowerCLI commands to change vNIC
Powered on pub, waited for services to reestablish
Powered off sub, ran VMware PowerCLI commands to change vNIC
Power on sub, waited for services to reestablish
Confirmed replication and tested voicemail box, call handlers, and MWI functionality
Confirmed all voicemail ports registered to CUCM cluster
Checked RTMT tool for any system error messages
Tested call handlers; tested MWI and messaging waiting indicators
Emailed licensing@cisco.com with generated license request from license server, Active SWSS/UCSS contract number, and PUT order number. Requested only Unity licenses only be migrated.

While this upgrade completed successfully without any downtime, I did run into several issues whose resolutions aren’t exactly satisfying.  If you’ve seen these or have insights into them, please leave a comment, I am sure I’m not the only one who has encountered these and would love to hear if others saw and dealt with them.

After the publisher rebooted when the install completed, the secondary’s web interface was a 404 and would not come back even after a reboot of the secondary. This forced the upgrade to be initiated via CLI for the secondary, but the issue went away in the new version.

Both servers after reboots established voicemail services (voicemail boxes available, call handlers working, etc…) in a matter of 10 – 15 minutes, but watching the RTMT tool, the CM platform services go into a CRITICAL DOWN state for about 30 mins, before eventually settling in an up state. Cisco Unified Serviceability was unreachable during this time, but voicemail services were working. No answer from TAC or forums as to anyone seeing this before and if this is going to cause long term issues.

A red DNS warning shows up in the OS administration and CLI for the servers even though DNS is working fine. Some forums suggest that I may have two entries in DNS forwarding, but that hasn’t panned out to be true. TAC suggests it’s a “cosmetic bug” and that since the functionality is clearly working, not to worry about it – yay?

Had to manually stop and restart replication after the power down/up for vNIC changes, hoping this was an after upgrade fluke, but time will tell.

*If you are looking for some VMware PowerCLI magic for those vNIC changes, I offer this – without support or guarantee, only that these commands did the trick for me.

PowerCLI C:\> Connect-VIServer [esxi IP address]
get-vm “servername”
get-vm “servername” | get-networkadapter
get-vm “servername” | get-networkadapter | set-networkadapter -type “vmxnet3”
Get-View -ViewType VirtualMachine -Filter @{“Name” = “servername”} | %{$_.reload()}

 

Published 1/8/2018

Updating DNS server IP addresses on Cisco voice servers

This post focuses on something that went right when making voice changes – shocking, I know!  Yes, I did open two different TAC cases before making the changes, and yes, I did have to talk to TAC once during the actual change process, but in my book, that’s a total win.

My mission, which the sys admin gave me no choice but to accept, was to change the DNS ip addresses each voice server was pointing to.  Now in sys admin world, this is hardly a big thing, but in voice world, no change is so small and insignificant that cannot be made intricately cumbersome to complete.

If you live in the CUCM 8.x universe, you are probably familiar now with the concept of a license mac. Changing any of the components that went into generating the precious license mac for the server invalidates all the pretty, pretty licenses you spent hours (likely even days) of your life trying to get in the first place. Changing primary DNS *is* one of those components in 8.x, so accepting your fate of needing to rehost licenses for ALL THE THINGS is step one in 8.x universe. Step two likely involves some heavy drinking.

But I currently live in 9.1.2 world – so some of this misery is offset.  The CUCM and Unity Connection (UNCX) teams at some point decided this whole invalidating-the-licenses-for-minor-changes deal was a suckfest, at least I’m assuming that was their thought process, so with version 9.x, changing primary DNS server ip address for CUCM and UNCX servers doesn’t upset the license mac gods. Cisco IM&P now leverages CUCM licensing in 9.x, so no rehosting required for those servers, either.

UCCX, though. Of course UCCX still cares. Because UCCX.

So enough background and onto the process – which is pretty straightforward considering it’s voice stuffs. Standard disclaimer, I put this list together from various calls with TAC asking for documentation and clarification on the process for each application and what to expect. Your mileage may vary, don’t take my word for it, always have a good backup, and certainly don’t blow your voice servers up. Check the docs, check with TAC. Note that for each application, the changes are made on the publisher/primary server first, then any subscribers or secondary servers.

For CUCM and UNCX servers:

-In the GUI of the License server, remove the CUCM or UNCX server instance from the License server.  Yes, I know the trepidation of deleting anything from a voice server – especially involving licensing, but it is strongly recommended to do so. If you forget or don’t do this, PROBABLY nothing will happen, according to my conversations with TAC.  But no way I’m taking that chance, you decide for yourself…

-At the CLI of the server, issue the following commands:

set network dns primary X.X.X.X
set network dns secondary X.X.X.X
show network eth0 – confirm the changes

-Add the instance back to the license server and synchronize. When adding the server back, remember it’s the OS username and password that you want to be using.

-Additional step for CUCM highly recommended by TAC: restart the Cisco Tomcat service at the CLI with the command utils service restart Cisco Tomcat

-This is also the process for stand alone license servers, but of course you don’t have to remove any instances from the license server or perform the Tomcat restart.

For IM&P servers:

-At the CLI of the server, issue the following commands:

set network dns primary X.X.X.X note you will still get an error message that rehosting is required, but I confirmed later with TAC that this is just a holdover error message from the 8.x days.
set network dns secondary X.X.X.X
show network eth0 – confirm the changes

For UCCX servers (note I have an HA environment):

-Take note of the current server license MAC, put it in a safe place.  I copied the contents of the license files to my desktop and took a screenshot of the current license configuration page. Because it’s voice and the paranoia with licensing runs deep.

-Sacrifice a chicken or two, and then at the CLI of the server, issue the following commands on each server:

set network dns primary X.X.X.X
set network dns secondary X.X.X.X
show network eth0 (confirm the changes)

-Reboot the primary, wait for what feels like an eternity for the primary to come back up and get it’s services started, then reboot the secondary.

-Take note of the new license mac and request a rehost, provide licensing with the new and the old license macs.

-Load the new license file and start happy dance.  Unless you hit an issue like I did and the new license won’t load. Then try another reboot of the pair, attempt license load once more with fingers more tightly crossed than before, and then proceed to happy dance.

Last, but not least, my good twitter friend and awesome voice guru Ryan Huff pointed out to me this Answer File Generator tool which can be used to predict your new license mac so you can request a rehost in advance. I decided that the 30 day grace period for UCCX would be enough for this project, but it’s fantastic to know that such a thing exists. Especially if you are going to be invalidating a lot of licenses, have a very small change window, and want to go ahead and get off grace period licensing as quickly as possible*.

 

*my rehosted license file for UCCX was generated in under 15 mins. Impressive. Still a PITA, but at least a quick PITA…

Published 09/09/2015

 

 

 

 

 

 

 

 

Changing your Unity Connection SMTP domain

Changing the SMTP domain on a Unity Connection server really isn’t that big of a deal, but as with all things voice, no change works as initially advertised.  Previously, I had never had cause to mess with the SMTP domain address, but recently one of the major cell providers quit delivering our voice mail message notifications to devices and, not surprisingly, users were none to happy about it.

My guess was that the carrier in question didn’t much like the format of the sender address, since it included a sub domain: unityconnection@myservername.mydomainname.com.  I quickly decided that changing the SMTP domain on the server would easily test that theory and seemed far less painful than opening a ticket with a large service provider*. I did open a TAC case just to see if there were any caveats in making this change I might want to be aware of. That’s a whole lot of voice experience talking…err, writing. The voice paranoia runs deep for a reason.

TAC indicated that not only was this a simple change as thought, but that only one service would have to be restarted – the Connection Conversation Manager, and that wasn’t to be a big deal. Well, finding that hard to believe**, I proceeded to make said change and found that there’s a little more to the story.

First – there are *three* services that have to be restarted, and since two of them are critical services, you experience a failover if you are running in HA. The system does warn you this will happen, and for what it’s worth I did not experience a loss of service doing this. Certainly don’t blame me, though, if you do have an outage and aren’t in a maintenance period when you attempt this change.

smtpUnity

 

Of course, you get to rinse and repeat on the secondary server in an HA environment. Personally, nothing about the warning prompt I got on the secondary server indicates this is not a big deal, but hey, maybe that’s just me…

smtp-connection conversation manager secondary

That being said, once the services were restarted, and my blood pressure returned to normal, I expected to see the SMTP domain updated and a happy dance to ensue.  Alas, that was not the case.  After consulting with TAC, and without the least bit of surprise whatsoever, I found that a reboot was “sometimes”, infer always, required.

This did fix my problem and the delayed happy dance was epic. And thankfully not recorded for the sake of my remaining pride.

 

Published 1/26/2015

*Almost all levels of hell are more pleasant than opening a case with a carrier. I suspect Satan actually admires the ingenuity of carriers and any future levels of hell are modeled on their expertise and innovation in human suffering.

**There is seemingly far more proof of the existence of the Loch Ness Monster, Big Foot, and the Abominable Snowman than of something involving voice being “easy”

Runt Post: Unity Connection 8.6 and SU5, there’s a .cop file for that

I ran into a fun* issue last week while upgrading Unity Connection from 8.6.(2a) to the latest patch SU5, and I would tell you to be sure to read the release notes on the subject, but as I found when calling TAC at 2am, the release notes don’t warn you about this particular issue.  See, after the upgrade, all those precious call handlers you painstakingly configured and tested over the past few years just don’t work. At all. Not even a little. In fact, the system just routes these calls to the Opening Greeting. Much to your frustration, you can see that the extensions are still defined on the call handlers just like before the upgrade, but after the patch the system rudely snubs your call handler configuration entirely.

As much as I am totally ruining the surprise of your discovering and experiencing this precious jewel of an error on your own and wondering what the heck to do about it, there is a .cop file that fixes the issue: ciscocm.cuc_86x_cdl3.cop.sgn (cco login required).   After uploading the file to both servers (assuming an HA environment), a reboot of the primary and then the secondary is all that’s left to do.

And people wonder why voice engineers drink…

 

Published 11/03/2014

*voice engineers have a slightly skewed definition of fun, precisely because of issues like this…

Note: there is a bug ID for this, CSCuq63776 (cco login also required) in case you are interested.  I have no idea if every upgrade that goes from 8.6(2a) will hit this particular bug, but at least now you won’t be surprised if you do…

Where’s my RTP?

As many of my readers already know, I recently shifted gears from a being a consultant focused on voice to a being network engineer focused on…well, everything. One of the first items on my list to tackle, however, put my years of voice training to good use.

A common issue voice engineers are faced with is one way or no way audio.  As a voice consultant, those types of issues almost always ended with my saying the line “that’s a network issue, you’ll need to follow up with *those* guys” because almost always one way or no way audio is a routing issue of some sort. Of course, now that I am again one of *those* guys that gets to fix the routing issue too, that line is not quite as charming.

Many people who do not do voice everyday are not aware of the fact that the actual audio payload of a phone call is RTP traffic sent directly between the two endpoints.  This is starkly different from the call signalling, which is typically between the endpoint and the Unified Communications Manager. Given that signalling and RTP can take differing paths, you can end up with phone calls that seem like they should work but don’t, and for better or worse, the other end cannot hear you screaming in frustration.

My most recent run in with this issue involved a new VPN connection setup and users unable to hear voicemail prompts when checking messages. Complicating the issue was the fact that “normal” phone calls – phone calls between the remote and main site, were working just fine. A closer look, however, easily explained why one was working and not the other.

A good place to start on a no audio issue like this is to make use of the Question Mark button located on most Cisco phones. In this case I was dealing with a 7960 (don’t judge me) and just clicking the Question Mark button twice pulled up stats for the conversation.  Most notably, I could see that the transmit packets were incrementing, but the receive packets were not*. Newer phone models like the 69/8900 series hide this information in some other menu which isn’t entirely obvious, but just check the documentation to see how to pull up phone call stats. You can see in the photo below the type of useful information you can get – note the Question Mark key is in the bottom right of the picture.

phonephoto

Having confirmed that the phone thought it was sending RTP packets, but clearly not receiving them, I decided it was a good time for a packet capture**. One of the things I love about Wireshark is how easy tracking down RTP conversations is in a .pcap file. To take the capture, simply plug into data port on the phone, launch Wireshark, start the capture, and make your test call. Then stop the capture, click on Telephony -> RTP -> Display All Streams. You will see the RTP conversations in the capture, choose the one you want, and then click Prepare a Filter to view just the RTP packets you are looking for. Be sure to wave your hand and mumble “these are the RTP packets you are looking for” while you do.

Once I had my filter applied I could see the two IP addresses that the RTP was attempting to flow between, one being the phone ip address and the other being the ip address of a secondary voicemail server, not located at the main site that the remote phones were typically calling.  I then tried to telnet to that voicemail server on port 5000 to monitor the voicemail port traffic but my connection was unsuccessful, reiterating that I had a routing issue somewhere.***

Capture

Now the conclusion to this episode is not nearly as exciting as the troubleshooting process, adding a route to the VPN configuration to the correct subnet was all that was required, but it serves as a great example of when a voice issue isn’t actually a voice issue, no matter how much it looks like one.

Published 5/25/2013

*Another item to rule out on these types of remote site setups is the possibility of a codec mismatch.  The default between sites is typically g729 and not all voicemail servers can tolerate that.  Using the Question Mark key can show you the codec that is being used and then you can investigate whether or not the proper transcoder resources are in place.

**Pretty much anytime is a good time for a packet capture.

***I did a write up on how you can monitor voice ports with Unity Connection 8.6 using telnet here

Unity Connection tidbits: forwarding voicemails by extension

Here’s a question I get asked quite often, “how do I forward a voicemail to someone by extension instead of by name?”

As it turns out, most people would rather have their fingernails pulled out one by one with a set of rusty pliers than to have to type in the first and/or last name of a user into a phone keypad.

Unity Connection does make a provision for extension dialing instead of named dialing when addressing/forwarding messages, but it is a global setting, so everyone will need be onboard*. I highly doubt this will be an issue…

Just mosey on over to System Settings > Advanced > Conversations and look for Disable Spelled Name Searches**. Should look something like this:

Disable Spelled Name Searches

Just check the box and you’re done!***  Yay you!

*See the comments section, Mike was kind enough to point out how to do this on an individual basis.  I’ve never been asked to do this for a single person, but it’s good to know it can be done.

**I desperately wish they would have called it something like Forward Voicemails by Extension (I know, clever, right?) to make it easier to find, but alas, now you know…

***Note this is not a feature you get to use with voice recognition, see documentation here: http://www.cisco.com/en/US/docs/voice_ip_comm/connection/8x/gui_reference/guide/8xcucgrg110.html#wp1168437

Published 4/10/2013

Tricks full of awesome: monitoring voicemail ports remotely

Here’s a neat trick I picked up in INE voice boot camp this week*: you can monitor Unity Connection ports from a telnet session.

Why would you want to do this?

Well, one, you are studying for the CCIE voice lab and you know that you won’t have access to the Unity Tools website to download the port monitor tool during the exam.

Two, you like cool and easy ways to do things and you think installing the port monitor tool is a terrific hassle**. I happen to fall into both categories.

Now to get this going – first be sure to turn on remote port monitoring in Unity Connection. Go to System Settings -> Advanced -> Conversations ->

Remote Port Monitor Parameter

You will see Enable Remote Port Status Monitor Output at the top if you’re in CUCM 7.x. Just check the box and enter the IP address of the router or host you will be initiating the telnet session from.

Then go to your router (or device of your preference) and issue this command:

telnet 10.10.10.10 5000 /source-int lo0 (note the IP address is the IP address of your Unity Connection server and the source-int should match the IP address you put in Unity Connection)

Here’s a nice sample of the output before I created a voicemail account for a user – you can see the caller’s extension and the voicemail pilot. You can also see the name of the voicemail port being used. Also note that when the system doesn’t find a voicemail box it plays the Opening Greeting instead.

Branch1#telnet 10.10.10.10 5000 /source-int lo0
Trying 10.10.10.10, 5000 … Open
CallData, 2, CallerId=1001, CalledId=8000, RedirectingId=, Origin=16, Reason=1, CallGuid=A409F619FAF34289B7F7092AAA6719A2, CallerName=My User, LastRedirectingId=, LastRedirectingReason=1024, PortDisplayName=PortGroup1-002
Application, 2, 1001, AttemptSignIn
State, 2, 1001, State – AttemptSignIn.cde!Dummy
State, 2, 1001, Event is [NULL]
Application, 2, 1001, PHTransfer
State, 2, 1001, State – PHTransfer.cde!LoadInfo
State, 2, 1001, Event is [TrueEvent]
Application, 2, 1001, PHGreeting
State, 2, 1001, State – PHGreeting.cde!PlayGreeting
Display, 2, 1001, Call answered if needed
Display, 2, 1001, Playing greeting for Call Handler: Opening Greeting
State, 2, 1001, Event is [HangupEvent]
State, 2, 1001, State – PHGreeting.cde!DoHangup
State, 2, 1001, Event is [HangupEvent]
Display, 2, 1001, Idle

Now here is the output after I created the voicemail box. You can see that it recognizes the user has a voicemail box and prompts the user to authenticate:

CallData, 2, CallerId=1001, CalledId=8000, RedirectingId=, Origin=16, Reason=1, CallGuid=EE22CD20CC714BF7A723D24A4153B306, CallerName=My User, LastRedirectingId=, LastRedirectingReason=1024, PortDisplayName=PortGroup1-002
Application, 2, 1001, AttemptSignIn
State, 2, 1001, State – AttemptSignIn.cde!Dummy
State, 2, 1001, Event is [NULL]
Application, 2, 1001, SubSignIn
Display, 2, 1001, Subscriber Sign-In
State, 2, 1001, State – SubSignIn.cde!AnswerPhone
State, 2, 1001, Event is [TrueEvent]
State, 2, 1001, State – SubSignIn.cde!AuthenticateUser
Application, 2, 1001, –>SubAuthenticate
State, 2, 1001, State – SubAuthenticate.cde!TryCounter
State, 2, 1001, Event is [NULL]
State, 2, 1001, State – SubAuthenticate.cde!GatherID
State, 2, 1001, Event is [FalseEvent]
State, 2, 1001, State – SubAuthenticate.cde!LoadSubscriberMinimalData
State, 2, 1001, Event is [NULL]
State, 2, 1001, State – SubAuthenticate.cde!GatherPIN
Application, 2, 1001, –>SubAuthenticatePW
State, 2, 1001, State – SubAuthenticatePW.cde!ValidatePwd
State, 2, 1001, Event is [HangupEvent]
Application, 2, 1001, <–SubAuthenticatePW
State, 2, 1001, Event is [HangupEvent]
Application, 2, 1001, <–SubAuthenticate
State, 2, 1001, Event is [HangupEvent]
Display, 2, 1001, Idle

And one final output I thought was interesting (yes I am a geek). Here is what it looks like when you enter the wrong pin, then enter the wrong ID and wrong pin when prompted again.

CallData, 2, CallerId=1001, CalledId=8000, RedirectingId=, Origin=16, Reason=1, CallGuid=8667665A30304959991818F0EE9C47B7, CallerName=My User, LastRedirectingId=, LastRedirectingReason=1024, PortDisplayName=PortGroup1-002
Application, 2, 1001, AttemptSignIn
State, 2, 1001, State – AttemptSignIn.cde!Dummy
State, 2, 1001, Event is [NULL]
Application, 2, 1001, SubSignIn
Display, 2, 1001, Subscriber Sign-In
State, 2, 1001, State – SubSignIn.cde!AnswerPhone
State, 2, 1001, Event is [TrueEvent]
State, 2, 1001, State – SubSignIn.cde!AuthenticateUser
Application, 2, 1001, –>SubAuthenticate
State, 2, 1001, State – SubAuthenticate.cde!TryCounter
State, 2, 1001, Event is [NULL]
State, 2, 1001, State – SubAuthenticate.cde!GatherID
State, 2, 1001, Event is [FalseEvent]
State, 2, 1001, State – SubAuthenticate.cde!LoadSubscriberMinimalData
State, 2, 1001, Event is [NULL]
State, 2, 1001, State – SubAuthenticate.cde!GatherPIN
Application, 2, 1001, –>SubAuthenticatePW
State, 2, 1001, State – SubAuthenticatePW.cde!ValidatePwd
Display, 2, 1001, Subscriber sign-in failed. Alias – myuser. Extension – 1001. Caller Id – 1001.
State, 2, 1001, Event is [FalseEvent]
Application, 2, 1001, <–SubAuthenticatePW
State, 2, 1001, Event is [FalseEvent]
State, 2, 1001, State – SubAuthenticate.cde!TryCounter
State, 2, 1001, Event is [NULL]
State, 2, 1001, State – SubAuthenticate.cde!GatherID
State, 2, 1001, Event is [NULL]
State, 2, 1001, State – SubAuthenticate.cde!GatherPIN
Application, 2, 1001, –>SubAuthenticatePW
State, 2, 1001, State – SubAuthenticatePW.cde!ValidatePwd
Display, 2, 1001, Subscriber sign-in failed. Alias – . Extension – – Not Available –. Caller Id – 1001.
State, 2, 1001, Event is [FalseEvent]
Application, 2, 1001, <–SubAuthenticatePW
State, 2, 1001, Event is [FalseEvent]
State, 2, 1001, State – SubAuthenticate.cde!TryCounter
State, 2, 1001, Event is [NULL]
State, 2, 1001, State – SubAuthenticate.cde!GatherID
State, 2, 1001, Event is [FalseEvent]
State, 2, 1001, State – SubAuthenticate.cde!LoadSubscriberMinimalData
State, 2, 1001, Event is [NULL]
State, 2, 1001, State – SubAuthenticate.cde!GatherPIN
Application, 2, 1001, –>SubAuthenticatePW
State, 2, 1001, State – SubAuthenticatePW.cde!ValidatePwd
Display, 2, 1001, Subscriber sign-in successful. Alias – . Extension – – Not Available -. Caller Id – 1001.
State, 2, 1001, Event is [Expired]
State, 2, 1001, State – SubAuthenticatePW.cde!ReturnExpired
State, 2, 1001, Event is [Expired]
Application, 2, 1001, <–SubAuthenticatePW
State, 2, 1001, Event is [Expired]
State, 2, 1001, State – SubAuthenticate.cde!ReturnExpired
State, 2, 1001, Event is [Expired]
Application, 2, 1001, <–SubAuthenticate
State, 2, 1001, Event is [Expired]
State, 2, 1001, State – SubSignIn.cde!AuthenticateUser_Expired
State, 2, 1001, Event is [NULL]
State, 2, 1001, State – SubSignIn.cde!RunSignInUtil
Application, 2, 1001, –>SubSignInUtil
State, 2, 1001, State – SubSignInUtil.cde!CheckAccountLocked
State, 2, 1001, Event is [FalseEvent]
State, 2, 1001, State – SubSignInUtil.cde!CheckExpiredAndNew
State, 2, 1001, Event is [NewUser]
State, 2, 1001, State – SubSignInUtil.cde!RunEnrollmentConv
Application, 2, 1001, –>SubEnrollment
State, 2, 1001, State – SubEnrollment.cde!PlayEnrollmentIntro
State, 2, 1001, Event is [TTStarEvent]
State, 2, 1001, State – SubEnrollment.cde!ReApplyRoutingRules
State, 2, 1001, Event is [NULL]
Application, 2, 1001, <–SubEnrollment
State, 2, 1001, Event is [NULL]
Application, 2, 1001, <–SubSignInUtil
State, 2, 1001, Event is [NULL]
Application, 2, 1001, AttemptSignIn
State, 2, 1001, State – AttemptSignIn.cde!Dummy
State, 2, 1001, Event is [NULL]
Application, 2, 1001, PHTransfer
State, 2, 1001, State – PHTransfer.cde!LoadInfo
State, 2, 1001, Event is [TrueEvent]
Application, 2, 1001, PHGreeting
State, 2, 1001, State – PHGreeting.cde!PlayGreeting
Display, 2, 1001, Call answered if needed
Display, 2, 1001, Playing greeting for Call Handler: Opening Greeting

*Credit for this tip goes to classmate Israel, thanks to him for bringing it to the class’s attention!

**Last time I tried to install the port monitor tool for Unity Connection on a Windows VM, it crashed the VM.  This may or may not be blamed on the tool itself, but after having to restore from snapshot, I am a bit wary of installing the tool.

Upgrades 101

I find the upgrade research process far more entertaining if you approach it like a less than ideal scavenger hunt – a poorly planned, slightly brutal game of talent and luck- mixed with just a pinch of despair.

In this example, which will not begin to cover all the ways/paths/routes you can go for an upgrade, we will take a common scenario and walk through the basic research process. This is more to outline the process and things you should be checking rather than trying to be a comprehensive list of links for upgrades.

I apologize in advance for all the links in this post that will inevitably become dead after a matter of time, but there’s only so much in the universe I can control. Also, I shouldn’t have to say this, but ALWAYS check the latest versions of the documentation – things change – but it’s highly unlikely I’ll modify this post to reflect it.  I’m just lazy that way.

Let’s say in our example you have 5 unified communications servers in your cluster and are looking to upgrade to the latest & greatest versions of each product:
2 Call Managers – MCS7825-H3 with 2×160 GB drives and 2 GB of memory running 7.1.3.20000-2
1 Unity server – MCS7825-H3 with 2×160 GB drives and 2 GB memory running Unity 8.0
2 UCCX servers – MCS7825-H3 with 2×160 GB drives and 2 GB memory running 7.0(1)SR05_Build504

Let’s start with determining the upgrade path for your Unity server:
In case you have been living under a rock, the reign of tyranny Unity has enjoyed over voice engineers has been given an official end date, so if you are looking to upgrade your voicemail server, Unity Connection is the way to go.  The breeze with which this product installs in comparison to it’s predecessor will make you want to kiss your mother-in-law and hug your neighbor’s yappy little dog.

Step one in the Unity Connection research process is to determine your hardware compatibility. In this case, you get to play the find-your-server-model on the Big Long List ‘o Compatibility for the version you want to go to – in this case I pulled up the compatibility list for 8.x of Unity Connection: http://www.cisco.com/en/US/docs/voice_ip_comm/connection/8x/supported_platforms/8xcucspl.html

In our current example, you can see that this model of server is supported for versions of Unity Connection 8.x.   Yay, you! Specifically, it’s supported for Platform Overlay 1 which requires 4 gig of RAM and 2 250 gig drives.

We will want to confirm that our proposed Unity Connection version is compatible with both our current version of CUCM and our proposed upgrade version of CUCM – this typically isn’t an issue since Unity Connection plays very nicely with almost all versions of CUCM, but definitely a check you want to make: http://www.cisco.com/en/US/docs/voice_ip_comm/connection/compatibility/matrix/cucsccpmtx.html

In the case of Unity Connection upgrades, you are going to want to look closely at the DiRT tool and the COBRAs tool to make the transition from the old to the new – there are excellent tutorials/instructions/information on this web page: http://ciscounitytools.com/

Moving on, let’s tackle the process of determining the upgrade path for CUCM:

Step one is once again to determine your hardware compatibility.  Find your model of server on this chart clearly constructed by evil forces seeking to wreak havoc on the universe: http://www.cisco.com/en/US/prod/collateral/voicesw/ps6790/ps5748/ps378/prod_brochure0900aecd8062a4f9.html

Note that if you actually want to see your server model AND the headers at the same time, you are just out of luck unless you have a really, really large monitor or can read really, really tiny font. I’ve been known to take a screen shot of the header row and a screen shot of my server model row and line the two up. The entire time chanting curses upon the chart creators.*

As you can see, our example server does support CUCM 8.6 but there is both an X and a (2).  If you look at the sub notations on this already freaking fabulous chart (note sarcasm font), you will see indicators that while this server is supported, there are certain memory and hard drive upgrade considerations.

In this case, our sample server meets the hard drive specs (160 GB drives), but is going to need an extra two gig of RAM to make the leap into hyperspace. One note here, be sure to watch out for servers that are supported only for bridged upgrades – this means you can take the server up to the latest version, but the only thing it’ll be good for is to take a backup that can be restored onto a supported server. That, and I’m sure it makes a really useful, if somewhat noisy, door stop.

Once you’ve confirmed hardware compatibility of your CUCM servers, you will need to check your upgrade path.  Even though ideally you would like to go directly to the latest and greatest CUCM version, unfortunately you may run across a you-can’t-get-there-from-here scenario – especially if you are looking at upgrading from versions of 6.x.  In our sample case we are looking to go from 7.1.3.20000-2 to 8.6.2.  Time to pull out the magic eight ball.  Nah, actually, just use this document:  http://www.cisco.com/en/US/docs/voice_ip_comm/cucm/compat/ccmcompmatr.html#wp373165

Note that you will likely need to know what your full version number translates to in SU-speak, in this case 7.1.3.20000-2 is actually 7.1(3a).  What you are looking for is your release in SU-speak under the Direct Upgrade column of the release you want to go to. If it’s not there, you are going to get to do a step upgrade – intermediate upgrades FTW! (sarcasm font again)

Be sure if you find yourself in this situation to pick a stable intermediate version.

The last piece of our CUCM cluster upgrade is UCCX (formerly IPCC):
Step one is, of course, to check the hardware compatibility, and there’s a doc for that: http://www.cisco.com/en/US/docs/voice_ip_comm/cust_contact/contact_center/crs/express_compatibility/matrix/crscomtx.pdf

This document may make your head spin a little, but first look for the version of UCCX you want to go with, in our case 8.5(1). Then check and see of your server makes the cut.

In our example case, we do make the cut – our version of UCCX 7.0(1)SR05_Build504 is even listed under supported upgrade paths- BUT looking at the document closely, UCCX 8.5(1) isn’t supported with our current CUCM 7.1.  Welcome back step upgrade.  You will need to keep your UCCX and CUCM versions compatible, so upgrading to UCCX to 8.02 then upgrading CUCM to 8.6, then upgrading UCCX to 8.5(1) will do the trick. And lead to excessive drinking.

Note, however, that our example CUCM version is not explicitly listed in the chart- only 7.1(3) and 7.1(3b) – so if it were me, I would confirm with Cisco support that 7.1(3a) indeed did support UCCX 8.02. Make no assumptions when it comes to the documentation- ever.

After you’ve survived all of this, I’d say your about 1/2 way through the research process.  Among other things, you will still need to review the release notes for each application, check for COP file requirements, check the prerequisites for each proposed version, and confirm phone firmware compatibility. You will also need to check third party application support for all your extra voice applications**, determine the order in which the upgrades need to happen, and review detail over again***.  You’ll also need to determine down times and back out options should upgrades not go as planned. Of course upgrades always go as planned, right? (sarcasm font at it’s finest)

This will get you started on your hardware check, help determine if you need to invest in new servers, and give you an idea of what version you should be targeting and what it’s going to take to get your hardware there.

*To say I hate this chart with the passion of a thousand suns is a woefully pathetic understatement. If it burned in the fires of Hades for all eternity, that wouldn’t be long enough.
**Be sure not to forget to check the compatibility for your Presence servers, your CER servers, your recording servers, your paging servers, your fax servers, your voice gateways, your legacy PBX, and anything else that integrates with CUCM
***Note that in some cases you may actually lose some functionality your users are dependent on, specifically I would mention the loss of built in Attendant Console when making the jump to CUCM 8 and above. Double check everything for gotchas and caveats, it’ll save your arse and your upgrade.

Follow that call!

In this entertaining episode of stump-the-voice engineer, users report a directory number is being routed through the Bermuda Triangle and landing in no-man’s voicemail land.  In fact, the mysterious voicemail greeting being heard by callers sounds like 3 minutes of someone pocket dialing their neighbor’s cat.

Proceeding to the standard fact collection, I find that the path of the call goes something like this:

PSTN -> gateway -> call manager-> directory number 1000 -> call forwarded to CTI route point 1001 -> 1001 call forwarded to Unity Connection voicemail -> voicemail greeting of unknown origin

If you’ve ever had to troubleshoot a similar issue, you probably already know where this is headed.

First test call: dial 1001, listen to greeting. Confirm correct greeting heard. Check.

Second test call: unforward 1000, let it go to voicemail. Surprise users by finding their cryptic voicemail greeting in record time.  Yep, users of this voicemail box had unintentionally recorded 3 minutes of scratchy noises and dead air, somehow managed to enable it as their standard voicemail greeting, and all without being the slight bit conscious of the process. Amazing.

While this of course boggles the mind, it doesn’t actually fix the real issue here – the fact that users want callers to hear the voicemail greeting for 1001 when 1000 is called and 1000 is forwarded to 1001.

For those not familiar, Unity Connection – in this particular case 8.x – has a couple of options on how to handle forwarded calls like the ones we are dealing with here.

The default behavior of Unity Connection is to use the first number when dealing with incoming calls that have been forwarded.  So in our case, even though 1001 is the directory number that forwarded to voicemail, the message for 1000 is being played out for the caller.  You can change this behavior globally by going to System Settings -> Advanced -> Conversations -> scrolling down toward the bottom and checking the Use Last (Rather than First) Redirecting Number for Routing Incoming Call box.

It’ll look something like this:

The problem with this solution is that some people, including the users in this case, actually like routing to voicemail boxes based on the first number called, so changing the behavior globally isn’t the best way to make friends and influence people.

Instead you can create a Forwarded Routing Rule in Unity Connection that changes the behavior just for a single the directory number – and only when that directory number is a redirecting number.

You will want to navigate to Call Routing -> Forwarded Routing Rules -> Add New. The rule looks something like this:

You will then set where you want the call to go. You can direct the call to a Call Handler, a Directory Handler, a Conversation, or to a User with a Voicemail Box.

Lastly, you will click Add New under Routing Rule Conditions to create the match statement.

Your match statement should look something like this:

Be sure to save your condition and your routing rule, and voilà! Incoming calls to voicemail containing the forwarding number you specified will now follow the rule you created. And there will be much rejoicing. If only in your own head.

Published 5/29/2012

Runt post: When voicemail is a dirty word

New deployments often require configuring a direct transfer to voicemail. Not too long ago @ifoam wrote this great piece on the steps involved in setting up this up: Transfer to Voicemail, which I recently referred to when I found that my configuration wasn’t working.

The article, however, confirmed my suspicions that I hadn’t missed any steps in the system configuration process, but when calls were transferred straight to the voicemail server, the user extensions weren’t coming along for the ride.

Fast forward several research minutes later to this obscurity, in particular the third entry by Randall White: https://supportforums.cisco.com/thread/2053902

Upon first reading, I found the fix too absurd to be likely, which I’m sure why Mr. White added the “no, I’m not joking” part.  The solution being proposed was the removal of the word “voicemail” from the alerting name of the CTI route point.

For those of us in voice, we’re rather familiar with what the alerting name controls, and no, it doesn’t usually have anything to do with this.  Alerting name shows up on phone displays and it’s generally one of those put-whatever-you-want-here-the-system-doesn’t-care fields. Except in this case it did. It cared a lot.

So instead of calling my CTI route point Direct To Voicemail – I changed it to Direct To VM.   Yep, that was it. I removed the offending vocabulary, quit infringing on the voicemail server’s sensitivities, and all was set right with the world.

And this is why voice engineers drink.

 

Publish Date: 2011/11/28