Upgrading UCCX 9.0(2) to 10.6(1)

If you’re wondering about upgrading UCCX from 9.0(2)SU2 to 10.6(1)SU2, and would like that information with a side of snark, then this is the post for you.

Fair warning, this is one of those your-mileage-may-vary entries, definitely do not take my notes as gospel because I promise *I* won’t be the one restoring your UCCX server from backup at 3am. Always read the release notes and upgrade guides in their entirety, skipping pages earns you nothing but (more) pain and suffering.

That being said, there’s a few important compatibility matrices you will want to check for this upgrade.

Pay close attention to whether your UCS can handle later versions of ESXi without UCS upgrades. For instance, 10.6 no longer supports ESXi 4.X, so you might have some pre-upgrade UCS work to be done before the real party gets started. If you are unsure about your UCS/ESXi version compatibility, try this link for checking.

UCCX 10.6 upgrade from 9.0(2) does require a .cop file, I used ciscouccx.refresh_upgrade_v1.9.cop.sgn.

Before beginning this process, I highly suggest taking a screenshot of your licensing, sacrificing a couple of chickens, and making sure you have ordered your upgrade license through the PUT tool. Voice engineering breeds much in the way of paranoia, so I also recommend downloading the scripts and prompts, just in case you get the additional fun of rebuilding the server from scratch. It should go without saying, but I’ll say it anyway, be sure to check that your backups have been running AND that they have been running successfully.

The order of operations goes something like this for an HA setup, note that there is no Finesse deployed in my setup, strictly CAD.

Confirm the primary is active and all services show IN SERVICE – don’t skip this, I’ve never tried to upgrade in an active failed over state, but I imagine it’s like crossing the streams and would end in much badness.
Install .cop file on the primary, reboot, grab coffee, wait for services to come back up
Install .cop file on the secondary, reboot, grab moar coffee, wait for services to come back up
Install upgrade file to primary, drive to a different county to get coffee, don’t panic when the server reboots during the installation, and do not reboot after install.
Install upgrade file to secondary, switch to vodka. Question life choices to get involved in voice engineering. Do not reboot after install.
Switch versions on the primary. There’s more than enough time at this stage to continue questioning your life choices. All of them.
Server (finally) boots to new version. Wait for services to start. The docs say this could take up to 30 minutes. Shouting profanities at the server will not shorten this interval significantly, but you’ll likely try anyway.
Log into the server, install license file, note the error message about OVA template issues. Shut down the server because seriously, who needs that kind of negativity in life? Or shutdown because you need to make changes to the OVA. Whichever…
Modify OVA template for RAM, OS, and vNIC changes*
Power on server, wait for services. Yes, again.
Switch versions on the secondary, repeat the process above, pour another glass of whatever is left.
Once the primary and secondary are both online with all services show IN SERVICE, check that replication status is good.
Run the client configuration tool and test your queues. Buy a lottery ticket if you haven’t had to call TAC,by this point.**

The above list is strictly an overview, but gives you a reasonable idea of what to expect during the upgrade. A whole lot of proper planning will result is a whole lot of waiting for things to happen, but not much else. An uneventful voice upgrade is an awesome voice upgrade.

*Things got interesting with this. During the planning stages, TAC sent me a link to the procedure for altering the OVA, related to Bug ID CSCut04158. This detailed the process to change the vNIC to vmxnet3. When I presented the process to my virtualization guru, he concluded it would not work for our configuration, we would have to use some PowerCLI magic instead. And by we, I mean he. He used PowerCLI magic, I just threw another chicken onto the alter. The code went something like get-vm MyServerName | get-networkadapter | set-networkadapter -type “vmxnet3” – use this suggestion at your own risk, I am not a virtualization expert, nor do I play one on TV.

**I will point out that in the upgrade guide, an error about unaligned partitions is called out as a potential issue – it sounded like a whole lot of no fun resulting in rebuilding from backup, and I was quite relieved I didn’t hit that one. Did I mention read the docs? Definitely do that…

Published 08/16/2016

PLAR, UCCX, Redirects, oh my…

Typically my UCCX (Unified Cisco Contact Center Express) stories involve much weeping and gnashing of teeth, but here’s one that (mostly) just left me scratching my head for a while.

I got a ticket reporting that a courtesy phone in a lobby was no longer working. After a little digging, I found that this phone was put in place to be a PLAR (private line automatic ringdown) phone, and when users picked up the handset they were greeted with an unfriendly fast busy instead of being courteously and automatically connected to the proper extension.

The phone in question was assigned its own Calling Search Space (CSS) and that CSS contained a single partition with a single translation pattern. This pattern translated “nothing” (meaning the user enters no digits) to an extension, but in typical network discovery fashion, I found that this extension pointed to yet another extension assigned to a different translation pattern and then that was pointed to a UCCX trigger. Yes, because complexity always makes the network run better.

First order of business, simplify. This extra translation didn’t seem likely to be the cause of the issue at hand, but it certainly wasn’t necessary so by pointing the PLAR translation the to UCCX trigger directly I made my life a whole lot easier, which I am a huge fan of doing.

Knowing what I know about UCCX and CUCM hand-offs, I honed in quickly on determining if the CSS of the translation pattern could “reach” the UCCX trigger. The device calling the UCCX trigger has to have access to the partition the trigger is in otherwise the call isn’t going through, and you’re going to be having a bad day.

In this case, we are actually talking about the CSS of the translation pattern and upon inspection the translation pattern was assigned a CSS that included the partition the UCCX trigger was in.

At this point, I needed to prove to myself that this was a UCCX issue and not a general CUCM issue, so I changed the destination on the translation pattern to my own extension – also in the same partition as the UCCX trigger. Worked like a charm. Definitely looking like UCCX weirdness then…

Quickly tired of grasping at straws, I took some logs from CUCM and threw them into Translator X and Notepad++, both of these tools are awesome and if you are not using them, you should be!

Here’s the line I kept seeing from my CUCM traces, indicating what I first suspected, a CSS issue:

10:03:35.086 |Digit analysis: potentialMatches=NoPotentialMatchesExist

CUCM could not find a potential match for the destination directory number in any partition in the CSS of the translation pattern. Which didn’t make sense. Which is why Contact Center so often makes me wanna cry.

I decided it was time to talk to my excellent friends over at TAC. After reviewing the configuration with the engineer, I was presented with Bug ID CSCso91760* which basically reports that instead of using the CSS on the translation pattern redirecting the call, the hand-off was using the CSS of the originating phone. In our case this meant the CSS being used was the PLAR CSS of the phone – which couldn’t reach anything except that single PLAR translation pattern. Lovely.

The fix was simple enough, though. In UCCX**, just go to the configuration page for the trigger and click the “Show More” button at the bottom. Then look for Calling Search Space for Redirect under the Directory Number Settings Section. Change the CSS from Default Calling Search Space to whatever CSS you need for the call to go through.

Here’s what you should be looking for in the trigger settings page, you can see that Default Calling Search Space is, well, the default:

Nothing further required, problem solved. Yay you! Now somebody pass the tissues…

Published July 17, 2013.

*CCO account required

**UCCX version in this case is 8.5, I was too lazy to research if earlier version have this same option box, if you have an earlier version and you don’t see this option, you may want an extra box of tissues.

Ways Contact Center Makes Me Cry, Chapter 3

Today’s troubleshooting episode is once again sponsored by UCCX and its numerous ways to bring grown engineers to their knees in despair. That might be a slight overstatement, but only by a smidge.

Quick summary of the scenario: an established agent needs to move to a new desk and take calls from the new location temporarily. I know what you are thinking, easy cheesy – IT unplugs her phone, totes it over to the new location, installs CAD on the new workstation, plugs everything in correctly, and leaves. User just shows up, logs in, and everything *works* – mission accomplished. IT is *that* good.

Yeah, that’s not what happened.

What if, for whatever perfectly legitimate reason that completely boggles the mind of any sane administrator, the user needs to have a brand new extension and a brand new phone added to this process. Still pretty easy, right? Well, kind of.

Being the diligent engineer you are, you remember to disassociate the user with her old phone, associate her with her new phone and extension- also selecting her new directory number for her UCCX extension in User Management. You even remember to add the new phone to the rmuser Application User to avoid those pesky JTAPI errors. Then you go ahead and reset the new phone a couple of times just for good measure.

You check resources in UCCX and you see the fruits of your awesome work reflected in the updated extension now showing for said user. You call the user and have her log in, expecting accolades and words of supreme gratitude and instead you are confronted with this beautiful piece of snark, beginning with the phrase The specified extension is not associated with the ID:

You promptly check with your system administrator as instructed and report back to yourself that you did follow the procedure for configuring an agent’s phone and UCCX can bite it.

Once out of tantrum state, here’s a more constructive approach to resolving this particular issue which I have seen more than a few times now:

Go ahead and check one more time, confirm that you really are seeing the updated extension in UCCX. I know, I know, of course you did this right, but the first time you don’t check, it’ll be something that simple that gets you.

Then, in Cisco Desktop Administrator, run a resync of users. Instructions vary on this one depending on the version. I can tell you that for the version in this little story, 7.0(1)Build168, the instructions are this:

1.       RDP/VNC to the UCCX server
2.       Go to Start> Programs> Cisco> Desktop> Admin
3.       Click on “Call Center1”
4.       Then go to Setup> “Synchronize Directory services”

You can perform these steps during production hours, but as always, proceed with caution if you have a lot of users or a server that is already overtaxed. Please don’t tell TAC that Amy said it was okay to do this and she crashed your server, I doubt they will be sympathetic. And they have my email address.

If you are reading this and trying to fix this issue at the same time, hopefully you won’t have to read any further. But if you have my kind of luck, here’s the next step you can try:

1. Stop and Restart the Desktop Sync service:

I have stopped and started this service during production hours several times, but once again, please see the above disclaimer. It’s *your* box, not mine…

And last, but not least, if you are still reading this for a solution, for one last hoorah, try this – NOT DURING PRODUCTION:

1. Stop and restart the UCCX node service. For the version mentioned here, see below, for later versions look to Cisco UCCX Serviceability for the Node Manager service:

This typically resolves the issue, much in the way that applying a hammer to make a very fine adjustment often does. Stopping and restarting this service will take your call center down, so submit your request for a maintenance window with a smile. Or be prepared to ask for forgiveness if you choose to just go for it. Again, I wouldn’t lead with the phrase, but Amy said…

Published: 06/25/2012

Upgrades 101

I find the upgrade research process far more entertaining if you approach it like a less than ideal scavenger hunt – a poorly planned, slightly brutal game of talent and luck- mixed with just a pinch of despair.

In this example, which will not begin to cover all the ways/paths/routes you can go for an upgrade, we will take a common scenario and walk through the basic research process. This is more to outline the process and things you should be checking rather than trying to be a comprehensive list of links for upgrades.

I apologize in advance for all the links in this post that will inevitably become dead after a matter of time, but there’s only so much in the universe I can control. Also, I shouldn’t have to say this, but ALWAYS check the latest versions of the documentation – things change – but it’s highly unlikely I’ll modify this post to reflect it. I’m just lazy that way.

Let’s say in our example you have 5 unified communications servers in your cluster and are looking to upgrade to the latest & greatest versions of each product:
2 Call Managers – MCS7825-H3 with 2×160 GB drives and 2 GB of memory running 7.1.3.20000-2
1 Unity server – MCS7825-H3 with 2×160 GB drives and 2 GB memory running Unity 8.0
2 UCCX servers – MCS7825-H3 with 2×160 GB drives and 2 GB memory running 7.0(1)SR05_Build504

Let’s start with determining the upgrade path for your Unity server:
In case you have been living under a rock, the reign of tyranny Unity has enjoyed over voice engineers has been given an official end date, so if you are looking to upgrade your voicemail server, Unity Connection is the way to go. The breeze with which this product installs in comparison to it’s predecessor will make you want to kiss your mother-in-law and hug your neighbor’s yappy little dog.

Step one in the Unity Connection research process is to determine your hardware compatibility. In this case, you get to play the find-your-server-model on the Big Long List ‘o Compatibility for the version you want to go to – in this case I pulled up the compatibility list for 8.x of Unity Connection: http://www.cisco.com/en/US/docs/voice_ip_comm/connection/8x/supported_platforms/8xcucspl.html

In our current example, you can see that this model of server is supported for versions of Unity Connection 8.x. Yay, you! Specifically, it’s supported for Platform Overlay 1 which requires 4 gig of RAM and 2 250 gig drives.

We will want to confirm that our proposed Unity Connection version is compatible with both our current version of CUCM and our proposed upgrade version of CUCM – this typically isn’t an issue since Unity Connection plays very nicely with almost all versions of CUCM, but definitely a check you want to make: http://www.cisco.com/en/US/docs/voice_ip_comm/connection/compatibility/matrix/cucsccpmtx.html

In the case of Unity Connection upgrades, you are going to want to look closely at the DiRT tool and the COBRAs tool to make the transition from the old to the new – there are excellent tutorials/instructions/information on this web page: http://ciscounitytools.com/

Moving on, let’s tackle the process of determining the upgrade path for CUCM:

Step one is once again to determine your hardware compatibility. Find your model of server on this chart clearly constructed by evil forces seeking to wreak havoc on the universe: http://www.cisco.com/en/US/prod/collateral/voicesw/ps6790/ps5748/ps378/prod_brochure0900aecd8062a4f9.html

Note that if you actually want to see your server model AND the headers at the same time, you are just out of luck unless you have a really, really large monitor or can read really, really tiny font. I’ve been known to take a screen shot of the header row and a screen shot of my server model row and line the two up. The entire time chanting curses upon the chart creators.*

As you can see, our example server does support CUCM 8.6 but there is both an X and a (2). If you look at the sub notations on this already freaking fabulous chart (note sarcasm font), you will see indicators that while this server is supported, there are certain memory and hard drive upgrade considerations.

In this case, our sample server meets the hard drive specs (160 GB drives), but is going to need an extra two gig of RAM to make the leap into hyperspace. One note here, be sure to watch out for servers that are supported only for bridged upgrades – this means you can take the server up to the latest version, but the only thing it’ll be good for is to take a backup that can be restored onto a supported server. That, and I’m sure it makes a really useful, if somewhat noisy, door stop.

Once you’ve confirmed hardware compatibility of your CUCM servers, you will need to check your upgrade path. Even though ideally you would like to go directly to the latest and greatest CUCM version, unfortunately you may run across a you-can’t-get-there-from-here scenario – especially if you are looking at upgrading from versions of 6.x. In our sample case we are looking to go from 7.1.3.20000-2 to 8.6.2. Time to pull out the magic eight ball. Nah, actually, just use this document: http://www.cisco.com/en/US/docs/voice_ip_comm/cucm/compat/ccmcompmatr.html#wp373165

Note that you will likely need to know what your full version number translates to in SU-speak, in this case 7.1.3.20000-2 is actually 7.1(3a). What you are looking for is your release in SU-speak under the Direct Upgrade column of the release you want to go to. If it’s not there, you are going to get to do a step upgrade – intermediate upgrades FTW! (sarcasm font again)

Be sure if you find yourself in this situation to pick a stable intermediate version.

The last piece of our CUCM cluster upgrade is UCCX (formerly IPCC):
Step one is, of course, to check the hardware compatibility, and there’s a doc for that: http://www.cisco.com/en/US/docs/voice_ip_comm/cust_contact/contact_center/crs/express_compatibility/matrix/crscomtx.pdf

This document may make your head spin a little, but first look for the version of UCCX you want to go with, in our case 8.5(1). Then check and see of your server makes the cut.

In our example case, we do make the cut – our version of UCCX 7.0(1)SR05_Build504 is even listed under supported upgrade paths- BUT looking at the document closely, UCCX 8.5(1) isn’t supported with our current CUCM 7.1. Welcome back step upgrade. You will need to keep your UCCX and CUCM versions compatible, so upgrading to UCCX to 8.02 then upgrading CUCM to 8.6, then upgrading UCCX to 8.5(1) will do the trick. And lead to excessive drinking.

Note, however, that our example CUCM version is not explicitly listed in the chart- only 7.1(3) and 7.1(3b) – so if it were me, I would confirm with Cisco support that 7.1(3a) indeed did support UCCX 8.02. Make no assumptions when it comes to the documentation- ever.

After you’ve survived all of this, I’d say your about 1/2 way through the research process. Among other things, you will still need to review the release notes for each application, check for COP file requirements, check the prerequisites for each proposed version, and confirm phone firmware compatibility. You will also need to check third party application support for all your extra voice applications**, determine the order in which the upgrades need to happen, and review detail over again***. You’ll also need to determine down times and back out options should upgrades not go as planned. Of course upgrades always go as planned, right? (sarcasm font at it’s finest)

This will get you started on your hardware check, help determine if you need to invest in new servers, and give you an idea of what version you should be targeting and what it’s going to take to get your hardware there.

*To say I hate this chart with the passion of a thousand suns is a woefully pathetic understatement. If it burned in the fires of Hades for all eternity, that wouldn’t be long enough.
**Be sure not to forget to check the compatibility for your Presence servers, your CER servers, your recording servers, your paging servers, your fax servers, your voice gateways, your legacy PBX, and anything else that integrates with CUCM
***Note that in some cases you may actually lose some functionality your users are dependent on, specifically I would mention the loss of built in Attendant Console when making the jump to CUCM 8 and above. Double check everything for gotchas and caveats, it’ll save your arse and your upgrade.

Ways Contact Center Makes Me Cry – Chapter 2

A useful skill set to have as an engineer is to recognize when communication between devices has broken down. Sometimes voice servers in particular need a kindly admin to step in and smooth over their cluster relationships; unfortunately, however, you’ll find they are just too ashamed to ask for help.

So here is a list of signs to help you tell when your UCCX servers have gone from perfectly compatible to particularly petty, but haven’t bothered to tell you about the upset.

Supervisors cannot listen to recordings.
In Cisco Agent Desktop an agent no longer see his/her call stats in the call log.
Historical reports gives an error message that no data exists for valid date ranges.
Historical reports tells you how exceptional you are with an Exceptional Error.*

If you see one or more of these symptoms, it’s likely one of your uppity UCCX servers has told the other he was taking his toys and going home. You can confirm this in one of several ways:

On versions below 8.x, check out the Data Control Center – both the Historical and Agent will likely display this gem of an error message: Error occurred while performing the operation. The cluster information and subscriber configuration does not match. The subscriber might be dropped (Please check SQL server log for more details).

On versions 8.x and above, you have a couple of options:

Go to UCCX Serviceability -> Tools -> Control Center – Network Services-> and see if the Cisco Unified CCX Database is showing as Out Of Service for either node.

Navigate to Tools -> Database Control Center -> Replication Servers and you will likely be greeted with this happy little declaration (in case you can’t read the message below, it starts with the phrase Publisher is DOWN, happy indeed):

So what do you do if your UCCX servers are indeed giving each other the silent treatment? Well, unfortunately, I’ve found that nothing short of rebooting cures this particular ailment. Sure you can click that “Reset Replication” button (after hours, of course) but it’s about as effective as hitting the elevator button over and over hoping that’ll make it come faster – really people, it doesn’t help! So just go ahead and plan that maintenance window to reboot the primary, followed by a reboot of the secondary.

But wait, there’s more!

If you noticed this issue because you are exceptional and your Historical Reports makes wild unsubstantiated claims that no data exists, just check out the solution under No Data Available in the Historical Reports in this link because there are a few more hoops to jump through:

http://www.cisco.com/en/US/products/sw/custcosw/ps1844/products_tech_note09186a0080b42524.shtml

Yep, you get to uncheck boxes and recheck boxes, and THEN reboot! The fun just keeps on coming!

So once your surly servers get an attitude adjustment in the form of a reboot, you’ll find that they have an amazing ability to forgive each other and everyone can now rejoice in cluster harmony.

*On one of my encounters with this issue I was lucky enough to generate not just an error, but an Exceptional Error. Still makes me laugh. Yep, still easily amused.

Published: 01/30/2012