Upgrading Unity Connection 9.1.2SU2 to 11.5.1SU2

Prologue: This post is intended as an informative prep document for those planning to upgrade Unity Connection (in an HA setup) from 9.1.2SU2 to 11.5.1SU2 using the OS platform upgrade and not PCD (because reasons…) – your upgrade mileage may vary and likely will. These are notes from my upgrade experience and while many of these steps are universal to the upgrade process, each upgrade is a unique snowflake. Do not use this post as a step by step How to Guide. You will be sorry, and I promise TAC won’t be impressed when you tell them that you followed a process from a blog, even one as awesome as mine.

So let’s get started on this upgrade for Unity Connection 9.1.2SU2 to 11.5.1SU2 – the best document to begin with is the Unity Connection upgrade guide found here. Your best bet is to read it through several times and to not be surprised when you can quote any particular paragraph and page number in your sleep.

Also, reviewing the release notes found here is a must.

Once you’ve read the docs and are starting to put together a plan, here’s some guidance on what that plan should entail:

Licensing. You will need some.  Be sure to order your upgrade through PUT and be sure that your licensing server can handle the version of Unity Connection you are going to.  You may need to upgrade your license server or add license definitions to your licensing server pull this off.

Installing a .cop file. Installing a .cop file for RSAv3 keys is listed in the upgrade guide. Digging deeper into this, if you are at version 9.1.2SU2 (AKA, this build contains the same fix as the .cop file.

To quote the release notes for this .cop file:

If you are already running CUCM release 10.x or higher you already have this fix and do not need to install this Cisco Options Package (COP) file. If you are running an
Engineering Special (ES)or Service Update (SU)that already contains CSCua88701you do not need to install this COP file. CSCua88701 is included in: and higher, and higher, and higher.

I skipped the installation of this .cop file per this information and also receiving no errors for the .cop results of the “run cuc preupgrade test” output run before the upgrade. For more details, check out this document.

Confirm ESXi support. You will want to be sure the ESXi version your virtual machine is on is supported for the version of Unity you want to be on, and that information lives here.

Determine OVA changes. This step requires checking here and comparing these requirements to your current virtual machine.You will likely need to change your virtual machine vNIC to VMXNET 3 as VMXNET3 is required. You may need to engage a little VMware PowerCLI magic to pull this off.*  Note, if you need more RAM or hard drive space, TAC says a rebuild of the server is required. I’ve heard rumors of engineers changing the memory but the official stance of support is not to touch that stuffs.

Determine your disaster rebuild strategy(s). I recommend downloading and installing COBRAS, getting it running, and exporting all your data from your voicemail servers. With this data, you can import into a fresh build if it comes to that. You should also be sure to have a good DRS backup, which when restored will have all your settings versus COBRAS which will require you to recreate some settings/features.

For me, the basic flow of the upgrade went something like:

Exported data using COBRAS
Confirmed available space on the common partition, need at least 25 GB – I did not have to change high and low water marks, refer to the guide if you need to.
Confirmed nightly backup
Verified voicemail ports registered pre-upgrade
Confirmed HA and replication status pre-upgrade
Ran “run cuc preupgrade test”
Chose not to use utils iothrottle disable command since 24/7 production environment – if I had used this command, after entering it, a reboot is required, re-enabling it post upgrade would also be another reboot
Installed upgrade to publisher, automatic switch version required
Waited for services to establish
Installed upgrade to subscriber, automatic switch version required
Waited for services to reestablish and replication to be good
Powered off pub, ran VMware PowerCLI commands to change vNIC
Powered on pub, waited for services to reestablish
Powered off sub, ran VMware PowerCLI commands to change vNIC
Power on sub, waited for services to reestablish
Confirmed replication and tested voicemail box, call handlers, and MWI functionality
Confirmed all voicemail ports registered to CUCM cluster
Checked RTMT tool for any system error messages
Tested call handlers; tested MWI and messaging waiting indicators
Emailed with generated license request from license server, Active SWSS/UCSS contract number, and PUT order number. Requested only Unity licenses only be migrated.

While this upgrade completed successfully without any downtime, I did run into several issues whose resolutions aren’t exactly satisfying.  If you’ve seen these or have insights into them, please leave a comment, I am sure I’m not the only one who has encountered these and would love to hear if others saw and dealt with them.

After the publisher rebooted when the install completed, the secondary’s web interface was a 404 and would not come back even after a reboot of the secondary. This forced the upgrade to be initiated via CLI for the secondary, but the issue went away in the new version.

Both servers after reboots established voicemail services (voicemail boxes available, call handlers working, etc…) in a matter of 10 – 15 minutes, but watching the RTMT tool, the CM platform services go into a CRITICAL DOWN state for about 30 mins, before eventually settling in an up state. Cisco Unified Serviceability was unreachable during this time, but voicemail services were working. No answer from TAC or forums as to anyone seeing this before and if this is going to cause long term issues.

A red DNS warning shows up in the OS administration and CLI for the servers even though DNS is working fine. Some forums suggest that I may have two entries in DNS forwarding, but that hasn’t panned out to be true. TAC suggests it’s a “cosmetic bug” and that since the functionality is clearly working, not to worry about it – yay?

Had to manually stop and restart replication after the power down/up for vNIC changes, hoping this was an after upgrade fluke, but time will tell.

*If you are looking for some VMware PowerCLI magic for those vNIC changes, I offer this – without support or guarantee, only that these commands did the trick for me.

PowerCLI C:\> Connect-VIServer [esxi IP address]
get-vm “servername”
get-vm “servername” | get-networkadapter
get-vm “servername” | get-networkadapter | set-networkadapter -type “vmxnet3”
Get-View -ViewType VirtualMachine -Filter @{“Name” = “servername”} | %{$_.reload()}


Published 1/8/2018


Posted by on 2018/01/08 in Unity Connection, Upgrades


Tags: , , , , ,

Converting Aruba IAPs to Campus APs for non-UAP access points

Welcome to a quick How To post for converting a shiny new Aruba Instant Access Point (IAP) into a shiny new Campus Access Point (controller based AP). This applies to access points that are not UAP.  For more info on the UAP process and which APs are UAP, check out this awesome post by @theitrebel.

Just power up your access point and once it’s booted, look for an “Instant” SSID.  Connect to this SSID, open a browser, and you will be redirected to a login page* for the controller.  The default login is admin/admin.

Next, click on the Maintenance link, and then navigate to the Convert tab. From the drop down, select Campus APs managed by a Mobility Controller.

Enter the IP or hostname of your controller and click the Convert Now button. Click the confirmation and the conversion will begin. Your AP will reboot, begin anew as a Campus AP, and join the controller you specified. This, of course, assumes you’ve got your controller configured correctly and licensing all in order.

If you’d like to see a short video demonstrating this process, I found this quick video posted by Sean Rynearson:

Bonus: If you have a bunch of IAPs to convert, I recommend reviewing this post on the subject, it’ll save you some time:


*if you are not automagically redirected when you join the SSID and open a browser window, try manually going to this URL:

Published 10/10/2017

Leave a comment

Posted by on 2017/10/10 in Aruba


Tags: , , ,

Seeing Tetration in action – NFD16

One of the highlights of Network Field Day 16 was a Cisco Tetration presentation by Tim Garner. Launched by Cisco last June, Tetration is a heavy lifting, data crunching platform that soaks up telemetry on all your network packets, uses machine learning algorithms on that data, and produces security policies templates based on the flow information received. This process gives engineers in-depth analytics, an impressive level of visibility, and supplies automagically crafted baseline security policies.  The latter truly shines when you are working with developers and application owners who have absolutely no clue what server needs to talk to what other server(s), much less what ports are required to do so securely.

With Tetration, you can use hardware sensors in the form of Nexus 9K switches with an -X in the SKU, or you can use software agents that can be installed just about anywhere. Or you can use a combination of both.  These sensors look at every single packet going in and out and generate telemetry packets that get shuffled off to Tetration where the real magic happens.

In addition to software agents and hardware sensors that natively generate Tetration metadata packets, you can also stream data from load balancers, firewalls, and other networking devices.  Some devices such as Citrix and F5 are natively supported, but others might take your doing a little work to get the data into a format that Tetration will accept – JSON being one of the acceptable formats.

Another interesting option for getting metadata into Tetration is the use of virtual machines set up as ERSPAN destinations.  Each VM can take in up to 40 gig of traffic, generate telemetry data for this traffic, and stream the data to the Tetration cluster.  Tetration can also take in NetFlow data using this VM method as a NetFlow receiver. NetFlow data is sampled though, so Tetration would not be seeing metadata on every packet as with the other options listed.

Once the data gets to the Tetration cluster, the snazzy machine learning algorithms built into the box start telling you cool things like what hosts are talking to what hosts and what “normal” network behavior looks like, and thereby, what abnormal network behavior would look like.

If your development servers should never be talking to your production servers, Tetration can tell you not only if that what’s happening now, but also if that behavior changes in the future.  Using a Kafka broker* you can have Tetration feed notifications to applications such as Splunk or Phantom, which can in turn communicate with hardware and software devices that perform actions such as host isolation when anomalous traffic is detected.

The automatic whitelists built by Tetration will require some care and feeding by an engineer. Importing policies from ACI is also an option as well. Tetration generated whitelists can be reviewed and tweaked, and an audit of what will be blocked when implementing or making policy changes is an excellent job preserving idea. Checking policies against the four to six months of network traffic data stored by the cluster gives you a good sense of what to expect when enforcement is actually turned on. That being said, you can also run your policies in audit mode for a few months to see what traffic hits the crafted policies.

If you want to see Tetration in action, I highly recommend this video below. The demo starts at about 16 minutes, but Tim Garner is such an excellent presenter, you’ll be glad you watched the whole thing.


*Kafka broker service was new to me, basically it’s a notification message bus, I used a few of these links below to get the idea:


Disclaimer: While Networking Field Day, which is sponsored by the companies that present, was very generous to invite me to this fantastic event and I am very grateful for it, my opinions are totally my own, as all redheads are far too stubborn to have it any other way.


Published 10/6/2017


Posted by on 2017/10/06 in Cisco, Network Field Day 16


Tags: , , , ,

Preserving and managing intent using Apstra AOS

Apstra’s Networking Field Day 16 presentations highlighted key issues engineers face every day.  The traditional ways of spinning up configurations and validating expected forwarding behavior falls short of the needs of networks of any size. For anyone who has encountered a network where the documentation was woefully outdated or practically non-existent, and whose design requirements and purpose were deduced purely from rumors and vague supposition, Apstra offers AOS and its Intent Store.

More than just configuration management, the idea of Apstra is to manage not just the building of consistent configurations abstracted from the specific hardware, but also to provide a controlled manner in which to address and manage network design revisions throughout the network’s life cycle.

Changing business needs impart necessary modifications to device dependencies and performance, Apstra addresses these revisions by maintaining a single source of truth – the documented intent* – and providing tools to validate this intent.  As Derick Winkworth said “it’s about moving the network [design] from being all in someone’s head” to making the design something consistent, tangible, and best of all, something that can be queried, and by extension, verified.

Under the covers, Apstra makes use of graph theory, and for those who’d rather not Google that, the upshot is nodes get added and relationships get tied to nodes.  The structure allows for a flexible schema that lends itself to ever-changing quantities of connections and also to new types of inter-dependencies between objects.

For example, Apstra added the ability to create links between network nodes and the applications that run through them.  This is done through some DevOps wizardary which this video highlights well, and the additional relationship mappings allow the network operator to query for application paths and diagnosis traffic flow issues.

For a short how-is-this-useful-to-me, I highly recommend this explanation by Damien Garros on using Apstra to shorten the time it takes to deploy crazy amounts of security zones, validate them, and monitor them. Snazzy stuff for any engineer who has ever faced a security auditor.


Disclaimer: While Networking Field Day, which is sponsored by the companies that present, was very generous to invite me to this fantastic event and I am very grateful for it, my opinions are totally my own, as all redheads are far too stubborn to have it any other way.


*Intent is all the buzz these days, back in my day we called it policy or design requirements. But I’ll try to avoid waving my arms and shouting get off my LAN… 🙂

Published 9/25/2017

1 Comment

Posted by on 2017/09/25 in Apstra, Network Field Day 16


Tags: , , , , ,

Minor version upgrades for Aruba controllers, version 6.5.x

If you tuned in for the last post, you’ll remember that in all the wireless mesh troubleshooting fun, a wireless controller upgrade was required.  Today’s post outlines the upgrade process from to with a primary and secondary controller. As always when dealing with upgrades, your mileage may vary. Never forget to read the release notes, and when in doubt, contact TAC.

As with any upgrade, the release notes often contain subtle details, and these are typically details that bite back if you miss them.  The notes for are pretty straightforward, but they do include exciting, not to be missed, caveats if upgrading from 6.4.x, as well as some solid tips for the upgrade.

The best practice advice in the notes includes steps such as confirming enough memory and storage space for the controller, making a backup of the configuration, and noting the number of active access points before upgrading. All of these suggestions make prudent sense and the commands to do so are listed in the guide.

You can use a FTP, TFTP, SCP, local file, or USB to do this upgrade, but the guide warns against using a Windows-based TFTP server. I used FileZilla FTP server.

Once you’ve downloaded the image file from Aruba and your pre-upgrade checklist is complete, navigate to Maintenance > Controller > Image Management -> Master Configuration.

Pick the file loading option you want to use for the upgrade, then fill in the required details for the transfer. Choose the non-boot partition for Partition to Upgrade. This will load the new software image to the inactive partition. If you are uncertain which one is the boot partition, look under Current Partition Usage, one partition will be marked **default boot**. You will want the other one.

Be sure that Reboot Controller after Upgrade is set to No, unless you have a single controller and eat danger for breakfast. Select Yes for Save Current Configuration Before Reboot, and click Upgrade.

At this point, you rinse and repeat for the other controller(s).  Once the controllers have the upgrade version loaded, you reboot the master, and simultaneously reboot the other controller. In voice upgrade world, you have been well trained to wait FOREVER for all the services to come back up on the primary before even considering a reboot of secondaries, but in Aruba wireless world, simultaneous is written in the guide. See excerpt below from the Release Notes available on the Aruba support site.

TAC did ease my anxiety over this simultaneous reboot thing by letting me know no harm would be caused if I wanted to wait for the master to come back online completely before proceeding.

After the controllers reboot and are back online, access points begin getting their new firmware and rebooting. Once the dust settles, you should end up with the same number of active APs as you noted in the beginning. Then it’s all testing and confirming access points are happy, clients are connecting, and that all is well your WiFi world.

Published 9/11/2017


Posted by on 2017/09/11 in Controller Upgrades, Wireless


Tags: , ,

Wireless troubleshooting – mesh access points stop accepting clients

Today’s topic initially spawned from mesh troubleshooting.  If you’ve worked with mesh much, you may have just thrown up a little and are frantically trying to close the browser tab that led you here, and that’s totally understandable.  For my voice engineer readers, mesh troubleshooting is one of the few things in the universe that can generate pain and suffering in levels akin to troubleshooting fax machines.

Given typically vague rumors and incomplete reports of intermittent connectivity issues at a mesh site, my amazing coworker was able to hone in on the root problem: various APs in the mesh intermittently stopped accepting clients on their 2.4 GHz radios. Being as 5 GHz was limited to back-haul traffic only, this was definitely wreaking some havoc on the overall deployment.

From the controller, disabling and re-enabling the radio on the 802.11g radio profile* for the affected APs served as a workaround while TAC was consulted. Mysteriously, other mesh deployment sites with the same model AP and code were not seeing this issue. As a note, these APs were all Aruba 274s and controller code version, but spoiler alert, the model AP wasn’t the issue.

Fast forward to TAC and some show/debugs commands later, the APs that randomly stopped accepting clients had enormous amounts of 2.4 radio resets, indicating this bug discussed here.

This issue affects not only 274s, but other models of access points. The bug does not appear to affect all of any model, just a small percentage of access points when it does show up.

If you think you might be experiencing this issue, take a look at the output of these commands and look for a crazy high number of radio resets.  How high?  Since the radio appears to be resetting practically every second, the radio reset number is noticeably and ridiculously large.**

show ap debug radio-stats ap-name NAME-OF-AP radio 0 advanced
show ap debug radio-stats ap-name NAME-OF-AP radio 1 advanced
show ap debug system-status ap-name NAME-OF-AP

The fix is in code versions or or later.  We landed and the issue looks to be properly squashed. The upgrade process, which I’ll outline in another brief post, was a simple and straightforward, and being a veteran of many a lengthy and challenging voice upgrades, I found this to be refreshingly delightful and far less sanity taxing.

* Enabling the radio can be done on the 802.11g radio profile, on the Basic tab.  Uncheck the Radio enable box, click Apply.  Check the Radio Enable box, click Apply.  These mesh APs each have their own AP specific profile and this action only affects the individual AP.  If your AP doesn’t have an AP specific profile, be sure to know what APs you are impacting when you do this. Also of note to this case, some experiencing this issue found disabling ARM provided temporary relief, but didn’t do the trick in this deployment, as ARM was already disabled and the issue still occurring.   

Radio enable

**Below is an example of the number of resets seen for one of the affected APs:

Interface Rx_pkts Rx_errors Rx drops Tx_pkts Tx_errors Tx_drops Resets
——— ——- ——— ——– ——- ——— ——– ——
wifi0 174210795 15727807 0 451900531 103 0 9
wifi1 9166677 133711103 0 32655175 842870 0 211094


Published 09/05/2017


Tags: , , , , , , ,

Cisco Live 2017, engineering awesomeness.

Spending a week with amazing engineers always ranks high on my list of reasons to attend Cisco Live every year.  The networking community and the behind the scenes work of the Cisco Live team make this event truly fantastic every year, and 2017 was a definitely a hit.

I especially enjoyed participating in Tech Field Day once again.  OpenGear presented Lighthouse 5 which focuses on automating setup and maintenance, leveraging new API goodness. OpenGear’s API aims to enhance scale of deployments, while streaming workflows.  I found it especially fun watching Slack be leveraged to enroll and communicate with the OpenGear device. Nerdy goodness I recommend checking out.

If you are looking for a monitoring solution, I highly recommend you check out this excellent PRTG demo by Benjamin Day of Paessler, who not only knows his stuff, but refuses to use even one Power Point slide for his Tech Field Day presentation.  The man is a genius. The PRTG notification enhancements, maps, and overall flexibility really stood out, definitely cool stuff. You won’t be sad you watched.

And in the final bit of Tech Field Day learning for me, NetApp’s presentation on their FlexPod SF solution took a room full of network engineers and captivated their attention on storage. I know it sounds hard to believe that network engineers could find a storage presentation fascinating, but Jeremiah Dooley managed to pull off this incredible feat, and I highly recommend checking out this session.  He covers all the important details of the FlexPod SF announcement, including the available architectures, in a way that makes network engineers forget that this is a solution focused on storing bits, and not just moving them.

The return of Engineering Deathmatch to Cisco Live featured several episodes with some of my fabulous (and lovingly voluntold for EDM) friends, who couldn’t be more amazing. I’m excited to check out the Engineering Deathmatch site as the episodes air over the coming weeks.

And lastly, my favorite annual tradition of Cisco Live wrap up blogging, the photo gallery of crazy, brilliant, hilarious engineers being remarkably phenomenal. I heart you all.

Published 07/04/2017


Posted by on 2017/07/04 in Cisco Live 2017


Tags: , , , , , ,