RSS

Upgrading Unity Connection 9.1.2SU2 to 11.5.1SU2

08 Jan

Prologue: This post is intended as an informative prep document for those planning to upgrade Unity Connection (in an HA setup) from 9.1.2SU2 to 11.5.1SU2 using the OS platform upgrade and not PCD (because reasons…) – your upgrade mileage may vary and likely will. These are notes from my upgrade experience and while many of these steps are universal to the upgrade process, each upgrade is a unique snowflake. Do not use this post as a step by step How to Guide. You will be sorry, and I promise TAC won’t be impressed when you tell them that you followed a process from a blog, even one as awesome as mine.

So let’s get started on this upgrade for Unity Connection 9.1.2SU2 to 11.5.1SU2 – the best document to begin with is the Unity Connection upgrade guide found here. Your best bet is to read it through several times and to not be surprised when you can quote any particular paragraph and page number in your sleep.

Also, reviewing the release notes found here is a must.

Once you’ve read the docs and are starting to put together a plan, here’s some guidance on what that plan should entail:

Licensing. You will need some.  Be sure to order your upgrade through PUT and be sure that your licensing server can handle the version of Unity Connection you are going to.  You may need to upgrade your license server or add license definitions to your licensing server pull this off.

Installing a .cop file. Installing a .cop file for RSAv3 keys is listed in the upgrade guide. Digging deeper into this, if you are at version 9.1.2SU2 (AKA 9.1.2.12901-3), this build contains the same fix as the .cop file.

To quote the release notes for this .cop file:

If you are already running CUCM release 10.x or higher you already have this fix and do not need to install this Cisco Options Package (COP) file. If you are running an
Engineering Special (ES)or Service Update (SU)that already contains CSCua88701you do not need to install this COP file. CSCua88701 is included in: 8.5.1.17123-1 and higher, 8.6.2.24122-1 and higher, 9.1.2.11018-1 and higher.

I skipped the installation of this .cop file per this information and also receiving no errors for the .cop results of the “run cuc preupgrade test” output run before the upgrade. For more details, check out this document.

Confirm ESXi support. You will want to be sure the ESXi version your virtual machine is on is supported for the version of Unity you want to be on, and that information lives here.

Determine OVA changes. This step requires checking here and comparing these requirements to your current virtual machine.You will likely need to change your virtual machine vNIC to VMXNET 3 as VMXNET3 is required. You may need to engage a little VMware PowerCLI magic to pull this off.*  Note, if you need more RAM or hard drive space, TAC says a rebuild of the server is required. I’ve heard rumors of engineers changing the memory but the official stance of support is not to touch that stuffs.

Determine your disaster rebuild strategy(s). I recommend downloading and installing COBRAS, getting it running, and exporting all your data from your voicemail servers. With this data, you can import into a fresh build if it comes to that. You should also be sure to have a good DRS backup, which when restored will have all your settings versus COBRAS which will require you to recreate some settings/features.

For me, the basic flow of the upgrade went something like:

Exported data using COBRAS
Confirmed available space on the common partition, need at least 25 GB – I did not have to change high and low water marks, refer to the guide if you need to.
Confirmed nightly backup
Verified voicemail ports registered pre-upgrade
Confirmed HA and replication status pre-upgrade
Ran “run cuc preupgrade test”
Chose not to use utils iothrottle disable command since 24/7 production environment – if I had used this command, after entering it, a reboot is required, re-enabling it post upgrade would also be another reboot
Installed upgrade to publisher, automatic switch version required
Waited for services to establish
Installed upgrade to subscriber, automatic switch version required
Waited for services to reestablish and replication to be good
Powered off pub, ran VMware PowerCLI commands to change vNIC
Powered on pub, waited for services to reestablish
Powered off sub, ran VMware PowerCLI commands to change vNIC
Power on sub, waited for services to reestablish
Confirmed replication and tested voicemail box, call handlers, and MWI functionality
Confirmed all voicemail ports registered to CUCM cluster
Checked RTMT tool for any system error messages
Tested call handlers; tested MWI and messaging waiting indicators
Emailed licensing@cisco.com with generated license request from license server, Active SWSS/UCSS contract number, and PUT order number. Requested only Unity licenses only be migrated.

While this upgrade completed successfully without any downtime, I did run into several issues whose resolutions aren’t exactly satisfying.  If you’ve seen these or have insights into them, please leave a comment, I am sure I’m not the only one who has encountered these and would love to hear if others saw and dealt with them.

After the publisher rebooted when the install completed, the secondary’s web interface was a 404 and would not come back even after a reboot of the secondary. This forced the upgrade to be initiated via CLI for the secondary, but the issue went away in the new version.

Both servers after reboots established voicemail services (voicemail boxes available, call handlers working, etc…) in a matter of 10 – 15 minutes, but watching the RTMT tool, the CM platform services go into a CRITICAL DOWN state for about 30 mins, before eventually settling in an up state. Cisco Unified Serviceability was unreachable during this time, but voicemail services were working. No answer from TAC or forums as to anyone seeing this before and if this is going to cause long term issues.

A red DNS warning shows up in the OS administration and CLI for the servers even though DNS is working fine. Some forums suggest that I may have two entries in DNS forwarding, but that hasn’t panned out to be true. TAC suggests it’s a “cosmetic bug” and that since the functionality is clearly working, not to worry about it – yay?

Had to manually stop and restart replication after the power down/up for vNIC changes, hoping this was an after upgrade fluke, but time will tell.

*If you are looking for some VMware PowerCLI magic for those vNIC changes, I offer this – without support or guarantee, only that these commands did the trick for me.

PowerCLI C:\> Connect-VIServer [esxi IP address]
get-vm “servername”
get-vm “servername” | get-networkadapter
get-vm “servername” | get-networkadapter | set-networkadapter -type “vmxnet3”
Get-View -ViewType VirtualMachine -Filter @{“Name” = “servername”} | %{$_.reload()}

 

Published 1/8/2018

Advertisements
 
11 Comments

Posted by on 2018/01/08 in Unity Connection, Upgrades

 

Tags: , , , , ,

11 responses to “Upgrading Unity Connection 9.1.2SU2 to 11.5.1SU2

  1. Arvind

    2018/01/08 at 16:25

    Ive always been reading your posts and never posted any comments so ill change this 2018.

    Great post and love the excitment you seem tl always have…i have pointed my team to your blog as well, hopefully they will learn as well…..question: PCD reasons?…no one likes pcd but ive kinda taken a shine to it:)

     
    • amyengineer

      2018/01/08 at 17:01

      Thanks for the positive feedback! On the PCD, it’s hard for me to get maintenance windows and I didn’t want to risk PCD failing and then needing to try again for the upgrade. I’ve heard and read that’s PCD is better than initial versions, but still not quite reliable. Our upgrades take a really long time (5 -6 hrs per server, TAC cannot explain why) so I wanted to be sure that we had the best chance of success the first go around. Also, I might be a bit old school, yelling at kids to get off my LAN… 😉

       
  2. Rob

    2018/01/08 at 16:56

    Love your posts Amy. Regarding the DNS warning. Good explanation here: http://benmorgan.com.au/blog/dns-unreachable-warning-cucm-version-11-x/

     
    • amyengineer

      2018/01/08 at 17:04

      I, no kidding, was reading your post during the upgrade! I even pointed it out to TAC. Still working with my server team to find any DNS issues like multiple records or PTR records that need to be deleted…

       
  3. Adam

    2018/01/08 at 18:00

    You’re a lot more cautious than I am.

    I just took snapshots for the CUCs before upgrade, wiped out the vNICs, added the correct vNICs, and away I went.

    I’ve pretty much found if you follow the docs, you won’t have any issues. Still, I created a CUC cluster at my house, then did the 11.5 upgrade on it. . .that’s where I found that you can just wipe the vNIC and replace it without issue. No PowerCLI or anything fancy like that. 9.1 found the new vNICs no problem, then I initiated the upgrade.

    If you want to cut your downtime for the upgrade, make sure all the extraneous stuff is at a minimum. So orphaned *Handlers, Users, Certs. . .make sure it’s cleaned up. When I went from 9.1SU2 to 11.5SU2, I had put in maintenance to do cleanup on the entire CUCM cluster (which included CUCM, IM&P, CUC) and made sure to run all the utilities for finding and fixing orphans, etc.

    I, too, have the DNS issue, but that’s because we have a stale PTR record that was created with a capital letter under 2003/2008R2. Those are almost impossible to delete, so reverse DNS doesn’t always resolve properly.

    Thankfully, you have 60 days to square away the licenses, so you don’t have to contact licensing before the upgrade. They were pretty speedy about getting me new licenses, though.

     
  4. Sillejo

    2018/01/09 at 18:55

    I’ve been out of the consulting admin game for a while, but the Time to complete variability on COBRAS was something that’s always shocked my clients. Does COBRAS now give u an inkling on how long it will take? If not, that question mark is an important guide. I did an upgrade to 9 and because the org was HEAVILY dependent on Vmail (plus they were so scared of the employee base they couldn’t ask them to clean out Vmail boxes for the upgrade) we spent the better part of 6 hours waiting for COBRAS to complete its backup (greetings and messages) Jeff L was still running Untiy Tools by himself back then and was awesome about emailing me back that weekend in almost real-time. Without that , I think they would have kicked me out mid upgrade.

     
  5. Adam Stauffer

    2018/01/12 at 14:39

    I had a WARNING: DNS Unreachable TAC case for a CUCM upgrade on 11.x in which the TAC engineer stated that starting in 11.X that the system does a reverse lookup but it also is case sensitive. So despite it resolving the DNS record the case mismatch was tripping my warning.

     
    • amyengineer

      2018/04/19 at 17:27

      That’s really good know! My issue ended up being with PTR records not having the FQDN. Easy fix once I found it.

       
  6. Brian

    2018/01/15 at 14:18

    Great post, missed the sacrificial chickens so it must not have been too bad.

     
  7. shpydah

    2018/02/05 at 15:46

    Hi Amy, long time reader and first time poster… I want to say Thank You for all the great write-ups of your process and experiences – they’ve been helpful to me on numerous occasions as both guides and “sanity checks” on weird/difficult stuff.

    I just completed the cluster upgrade of both CUCM and CUC from 10.5.2 to 11.5.1 on CUCM and 9.1.2 to 11.5.1 on CUC this weekend, and because of good prep, and double checking my ducks knew their place, all went very smoothly.

    I experienced the following minor issues:
    – I also had 404 response from the subscriber web GUI subsequent to the Publisher install, and like you, I had to initiate the upgrade on the sub from CLI – but it went fine after that
    – After upgrading the Pub (CUCM), a previously undetected, expired CAPF cert on the sub started dropping events into RTMT – the expiration was 1/7/2017 and so not sure why it hadn’t cried sooner; this added significant time to the flow as regenerating CAPF certs (among others) necessitates a phone soft reboot – cluster wide
    – Ran into bug CSCul63736 after upgrade but it’s no biggie – “cosmetic” bug regarding failed DR backup being reported even though the back up completes successfully
    – Like poster above, I was able to blow out the “flexible” adapter and add the VMXNET3 on the VMs without the use of PowerCLI – although I did have that installed and ready on each management VM in case it was needed
    – Regarding “iothrottle disable”, I had a long maintenance window for this and went ahead and used the command – and planned for the reboot to reenable once I was done – however I found that the iothrottle setting came back ENABLED after the first reboot subsequent to upgrade – so that’s neat
    – After upgrade was complete, I found that RTMT cannot poll one of the CUCM subs when connected to the Pub – to pull stuff like log data or traces I have to connect RTMT directly to that sub – I still haven’t figured that one out yet but I’ll get it. The message I get from RTMT when I try to pull logs is “Could not connect to ‘Server: cm-sub’ Please check and correct” Yadda yadda stuff about NAT and FQDNs – I’ll dig into it

    Overall, super smooth upgrade, gross operations were completed in about 16 hours; 3 nodes, 6 instances, and 2 countries (Toronto office!) It’s enormously helpful and reassuring to have blogs like yours, so thanks a bunch!

     

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

w

Connecting to %s

 
%d bloggers like this: