Bye, bye 2800s…

At the end of this month, the long beloved 2800 series voice gateways go end of support.  If you find yourself finally getting around to replacing these boxes with the 4000 series voice gateways, you’ll be happy to hear that transitioning PRIs from one to the other is relatively painless for voice work*.

I found it did take me a few minutes to hone in on the shiny new syntax for the clocking commands, so here’s a quick overview of what you’ll need to know.

We’re all pretty used to the standard “network clock participate, blah, blah, blah…” command paired with the “network clock select something-or-other…” command to keep those slip seconds on the T1 controller away.**

With the 4Ks, you need something similar, but of course, they couldn’t just leave the syntax the same. Instead you need to use something like this, of course using the slot numbers that your T1 card is installed in:

network-clock synchronization automatic
network-clock input-source 1 controller t1 0/1/0

[a previous version of this post had this command network-clock synchronization participate, but a very smart TAC engineer alerted me to bug ID CSCvb01800 which says to take the participate command out even though some Cisco docs on the subject include it.]

And don’t forget to add “clock source line primary” on the controller port – you didn’t typically need to explicitly set this on the 2800s/2900s, but apparently the 4Ks need more hand holding and direct instructions.

controller T1 0/1/0
 framing esf
 clock source line primary
 linecode b8zs
 cablelength long 0db
 pri-group timeslots 1-24

When you get this right, you should see some good news similar to this scroll across the screen – feel free to do a little happy dance:

*Sep 9 21:11:01.139: %NETCLK-6-SRC_ADD: Synchronization source T1 0/1/0 is added to T0 selection process.

And you can check your T1 clocking information when you bring the circuit up with this handy little command:

#show network-clock sync
Symbols:     En – Enable, Dis – Disable, Adis – Admin Disable
             NA – Not Applicable
             *  – Synchronization source selected
             #  – Synchronization source force selected
             &  – Synchronization source manually switched

Automatic selection process : Enable
Equipment Clock : 2048 (EEC-Option1)
Clock Mode : QL-Disable
ESMC : Disabled
SSM Option : 1
T0 : T1 0/1/0
Hold-off (global) : 300 ms
Wait-to-restore (global) : 300 sec
Tsm Delay : 180 ms
Revertive : No

Nominated Interfaces

 Interface            SigType     Mode/QL      Prio  QL_IN  ESMC Tx  ESMC Rx
 Internal             NA          NA/Dis       251   QL-SEC    NA        NA       
*T1 0/1/0             NA          NA/Dis       1     QL-SEC    NA        NA

After cutting over to the new gateway, you can check for slip seconds on the line using the old standard “show controller T1 0/1/0” – but be sure you clear the counters after plugging in the circuit, since there are always a few slip seconds reported when first plugging in the circuit.

After resetting the counters, check that the slips stay at zero.

#show controller t1 0/1/0
T1 0/1/0 is up.
  Applique type is Channelized T1
  Cablelength is long gain36 0db
  No alarms detected.
  alarm-trigger is not set
  Soaking time: 3, Clearance time: 10
  AIS State:Clear  LOS State:Clear  LOF State:Clear
  Framing is ESF, Line Code is B8ZS, Clock Source is Line Primary.
  Data in current interval (566 seconds elapsed):
     0 Line Code Violations, 0 Path Code Violations
     0 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins
     0 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs

[output truncated]

Bonus material – the other strange syntax issue I hit with the 4Ks was bug ID CSCup86596 and the command “isdn incoming-voice voice” command wouldn’t take under the signaling channel configuration – the workaround is epic in its vagueness, noting that “the functionality is there and will work” even though you cannot enter the command.  So, yeah, typical voice shenanigans…


More bonus material – I was made aware by the previously mentioned very smart TAC engineer, that if you are looking into RTP-NTE (RFC 2833) DTMF issues on a 4K, you are going to need pcaps, the debug voip rtp sess name doesn’t do the trick.  I haven’t had to face this one yet, but hopefully that little bit of information will save you some time if you do.


*voice is pain, any one who tells you otherwise is trying to sell you something  😉

**to set clocking for PRIs on 2800s/2900s, typically you use something like these two commands, depending on which slot the WIC card is plugged into. If you are seeing slip errors, you should check for these commands: 

network-clock-participate wic 1
network-clock-select 1 T1 0/1/0

Published 10/21/2016





Tags: , , , , , , , , , , ,

Upgrading UCCX 9.0(2) to 10.6(1)

If you’re wondering about upgrading UCCX from 9.0(2)SU2 to 10.6(1)SU2, and would like that information with a side of snark, then this is the post for you.

Fair warning, this is one of those your-mileage-may-vary entries, definitely do not take my notes as gospel because I promise *I* won’t be the one restoring your UCCX server from backup at 3am.  Always read the release notes and upgrade guides in their entirety, skipping pages earns you nothing but (more) pain and suffering.

That being said, there’s a few important compatibility matrices you will want to check for this upgrade.

Pay close attention to whether your UCS can handle later versions of ESXi without UCS upgrades. For instance, 10.6 no longer supports ESXi 4.X, so you might have some pre-upgrade UCS work to be done before the real party gets started. If you are unsure about your UCS/ESXi version compatibility, try this link for checking.

UCCX 10.6 upgrade from 9.0(2) does require a .cop file, I used ciscouccx.refresh_upgrade_v1.9.cop.sgn.

Before beginning this process, I highly suggest taking a screenshot of your licensing, sacrificing a couple of chickens, and making sure you have ordered your upgrade license through the PUT tool. Voice engineering breeds much in the way of paranoia, so I also recommend downloading the scripts and prompts, just in case you get the additional fun of rebuilding the server from scratch. It should go without saying, but I’ll say it anyway, be sure to check that your backups have been running AND that they have been running successfully.

The order of operations goes something like this for an HA setup, note that there is no Finesse deployed in my setup, strictly CAD.

  • Confirm the primary is active and all services show IN SERVICE – don’t skip this, I’ve never tried to upgrade in an active failed over state, but I imagine it’s like crossing the streams and would end in much badness.
  • Install .cop file on the primary, reboot, grab coffee, wait for services to come back up
  • Install .cop file on the secondary, reboot, grab moar coffee, wait for services to come back up
  • Install upgrade file to primary, drive to a different county to get coffee, don’t panic when the server reboots during the installation, and do not reboot after install.
  • Install upgrade file to secondary, switch to vodka. Question life choices to get involved in voice engineering.  Do not reboot after install.
  • Switch versions on the primary. There’s more than enough time at this stage to continue questioning your life choices. All of them.
  • Server (finally) boots to new version. Wait for services to start. The docs say this could take up to 30 minutes. Shouting profanities at the server will not shorten this interval significantly, but you’ll likely try anyway.
  • Log into the server, install license file, note the error message about OVA template issues. Shut down the server because seriously, who needs that kind of negativity in life? Or shutdown because you need to make changes to the OVA. Whichever…
  • Modify OVA template for RAM, OS, and vNIC changes*
  • Power on server, wait for services. Yes, again.
  • Switch versions on the secondary, repeat the process above, pour another glass of whatever is left.
  • Once the primary and secondary are both online with all services show IN SERVICE, check that replication status is good. 
  • Run the client configuration tool and test your queues.  Buy a lottery ticket if you haven’t had to call TAC,by this point.**

The above list is strictly an overview, but gives you a reasonable idea of what to expect during the upgrade. A whole lot of proper planning will result is a whole lot of waiting for things to happen, but not much else.  An uneventful voice upgrade is an awesome voice upgrade.


*Things got interesting with this. During the planning stages, TAC sent me a link to the procedure for altering the OVA, related to Bug ID CSCut04158. This detailed the process to change the vNIC to vmxnet3. When I presented the process to my virtualization guru, he concluded it would not work for our configuration, we would have to use some PowerCLI magic instead. And by we, I mean he. He used PowerCLI magic, I just threw another chicken onto the alter. The code went something like get-vm MyServerName | get-networkadapter | set-networkadapter -type “vmxnet3” – use this suggestion at your own risk, I am not a virtualization expert, nor do I play one on TV. 

**I will point out that in the upgrade guide, an error about unaligned partitions is called out as a potential issue – it sounded like a whole lot of no fun resulting in rebuilding from backup, and I was quite relieved I didn’t hit that one. Did I mention read the docs?  Definitely do that…

Published 08/16/2016











Posted by on 2016/08/16 in UCCX, Upgrades, Upgrades


Tags: , , , , ,

2016 Cisco Live, US – Geeks for life.

Putting together a wrap up post on Cisco Live US always makes me smile, and 2016 is no exception.  As many of you know, this CLUS marked 5 years since Tom and a small group of engineers first bonded over networking nerdiness and an addiction to 140 characters.

We’ve followed and helped each other through upgrades, outages, career changes, certifications, and a plethora of engineering challenges. We’ve commiserated with the suck that sometimes is our jobs, we’ve championed the hard-fought successes of our peers, and as a bonus, we’ve managed to provide an ever flowing stream of hilarious commentary along the way.

I’m thrilled to be part of a networking community that just keeps growing larger, and I’m continually impressed with the engineering talent this group represents.  I’m also pleased to have been part of Tech Field Day at CLUS as well, which continues to do an amazing job bringing great content and engineers together.

Just check out this awesomeness. Could there be anything better? No, no there could not.


Tags: ,

8841s and line selection woes…

Just a quick post in case there is at least one other person in the world that didn’t know this particular trick for the 8800* series phones that always go off hook on the last line used.  In case you weren’t aware that these phones had this behavior, don’t worry, as soon as you deploy one to a receptionist that handles more than a single phone line, you WILL be made aware. You may not survive the encounter, but you will know…

Should the user let you live long enough to research a solution, you will find that TAC doesn’t offer you any hope in restoring the requisite amount of receptionist happiness to keep oxygen flowing to your lungs. TAC will politely inform you that the phone is operating as designed and that you might want to buy the receptionist some flowers instead.  Okay, maybe that last part was my idea…

Technically TAC is right, you cannot force the phone back to selecting the primary line after hanging up a call that was answered or placed on any of the other lines. However, as my awesomely brilliant coworker discovered, you can change the behavior of the phone so that when calls come in they are DISPLAYED on the primary line.

This way, when the user hangs ups, the phone still has the primary line selected, even if the last call received was for one of the other lines on the phone. This mimics a behavior users have come to know and love of the older generation of phones, a behavior users who manage multiple lines desperately want back.

You can set this option on the phone configuration page, just search for Show All Calls and change the feature to Enabled.

Show All Calls on Primary Line

This won’t make the remembering-the-last-line-selected-for-off-hook behavior go away, but it does make its impact far less noticeable to end users who might otherwise want to plot your demise.

* In this case we were working with 8841 phones, but my coworker has used this trick in the past on 9971 phones, so I’d be inclined to believe this would work on pretty much any SIP phone that has this behavior. 

Published 04/11/2016


Tags: , , , , , ,

Tom’s Corner – Cisco Live Twitter community 5 years later…

If you’re an annual attender of Cisco Live like I am, you might be noticing there’s some pretty cool stuff taking shape for 2016.

This year will be the 5th anniversary of Tom’s Corner – the first ad hoc gathering location for us geeky, twittering engineers*.

Hard to believe, but in 2011 we were just a smallish group of network nerds, some with quite a bit more hair in those days, and of course, the one random photo-bomber guy on the left. Still nobody knows who that guy was…

The Twitter Group 2011

The amazing amount of talent represented and the camaraderie of the network engineering community is what makes Cisco Live the place to be.

This year I’m also looking forward to witnessing some epic Engineering Deathmatch action. If you aren’t familiar with Engineering Deathmatch you gotta watch this video.  You won’t be disappointed.

And if that weren’t enough Cisco Live goodness, I gotta admit I’m pretty excited about hearing Keyser Söze, err Kevin Spacey, speak.

Other useful details for attendees:

  • The scheduler will be available on  April 26th for NetVets, May 3rd for the rest of us.
  • The band this year is Maroon 5, and the CAE is going to be at the shiny new T Mobile Arena
  • Newbies to Cisco Live can get a super cool mentor to help them figure the conference thing out, just sign up here
  • Tech Field Day will be at Cisco Live this year as well, look for fabulous content from them. If you’re interested in being a delegate for this or any event, apply here
  • Awesome blogs to watch for new Cisco Live content and to read past posts for excellent conference advice: @fryguy_pa, @scottm32768, @mrtugs@net_introvert, @networkingnerd, @someclown, @aconaway, @danieldibswe, @gingmarCisco Live Attendee Blogs
  • The annual twitter list is out, sign up here!

*Tom’s corner even had it’s own check in on Four Square. In case there was any doubt we were and still are total nerds…

Published 04/11/2016

Leave a comment

Posted by on 2016/04/11 in Cisco Live


Tags: , ,

Cisco Live Europe: NBASE-T progress report

The Ether

Robert Metcalfe –May 22nd, 1973 – first memo describing Ethernet. Photo Credit:

If you ever get a chance to watch Peter Jones present on a topic, I highly encourage you to take it! He’s passionate about multigigabit and the evolution of Ethernet technology, and it shows.

In this CLEUR Tech Field Day Extra session, Peter outlines significant progress that has been made on the 2.5G and 5G standardization process, which is pretty impressive considering the NBASE-T Alliance was only formed in October of 2014.

In addition to announcing some Cisco products that will be capable of multigigabit magic, Peter also lays out some additional interesting use cases, besides just the hotly debated timeline of the impending 802.11ac Wave 2 data rate apocalypse.

A few highlights from the product announcement pieces of the presentation, be sure to consult relevant data sheets for accuracy, caveats, and any insomnia issues you might have:

  • 48 port blade for the 4500E, 12 ports will do multigigabit
  • 3850 24 or 48 port switch, half of the ports will do multigigabit; note this is an entirely new SKU but stacking is compatible with previous 3850s
  • 3560-CX compact switch, two ports will support the multigigabit
  • All multigigabit ports are supporting UPOE (60 Watts)
  • No SFP+ available – Peter talks briefly about the heat challenges around this form factor, interesting geeky stuff
  • 3800 series AP

While I agree that there’s certainly some questions as to which devices and their data rates will bring current copper infrastructure to its knees, realistically getting more mileage out of already installed cabling makes a good kind of sense. Hardware refresh cycles are generally much shorter intervals than building wiring replacement schedule. Finding ways to cope with aging copper cabling and eking out a few more years of use from such a significant investment is a reality for engineers.

To leverage 2.5G or 5G magic, both devices – what’s getting plugged in and what’s being plugged into – will need to support the multigigabit technology.  Coordinating equipment refresh cycles and budget to make this happen could be quite the challenge as well.  Some number crunching would be appropriate to determine cost effectiveness for your own environment, but multigigabit Ethernet could be that additional tool in the belt when it comes to maximizing cabling investments already made.

Also of nerdy note, Peter’s short discussion of how the multigigabit magic filters out the noise using advanced coding and signal processing is quite fascinating. These brief comments led to me down a rabbit hole and to this interesting article,Crosstalk problems are back, which discusses the issues of noise when dealing with high data rates across copper wiring.

I highly recommend checking out these posts from other awesome engineer bloggers, good stuff:

And for more learning on the topic, definitely check out these:

Fundamentals of NBASE-T


Standard Disclaimer: Tech Field Day covered my expenses at Cisco Live Europe, but I am a redhead, any thought that my opinions could be bought or dictated is just crazy talk. 


Tags: , , , , ,

Updating DNS server IP addresses on Cisco voice servers

This post focuses on something that went right when making voice changes – shocking, I know!  Yes, I did open two different TAC cases before making the changes, and yes, I did have to talk to TAC once during the actual change process, but in my book, that’s a total win.

My mission, which the sys admin gave me no choice but to accept, was to change the DNS ip addresses each voice server was pointing to.  Now in sys admin world, this is hardly a big thing, but in voice world, no change is so small and insignificant that cannot be made intricately cumbersome to complete.

If you live in the CUCM 8.x universe, you are probably familiar now with the concept of a license mac. Changing any of the components that went into generating the precious license mac for the server invalidates all the pretty, pretty licenses you spent hours (likely even days) of your life trying to get in the first place. Changing primary DNS *is* one of those components in 8.x, so accepting your fate of needing to rehost licenses for ALL THE THINGS is step one in 8.x universe. Step two likely involves some heavy drinking.

But I currently live in 9.1.2 world – so some of this misery is offset.  The CUCM and Unity Connection (UNCX) teams at some point decided this whole invalidating-the-licenses-for-minor-changes deal was a suckfest, at least I’m assuming that was their thought process, so with version 9.x, changing primary DNS server ip address for CUCM and UNCX servers doesn’t upset the license mac gods. Cisco IM&P now leverages CUCM licensing in 9.x, so no rehosting required for those servers, either.

UCCX, though. Of course UCCX still cares. Because UCCX.

So enough background and onto the process – which is pretty straightforward considering it’s voice stuffs. Standard disclaimer, I put this list together from various calls with TAC asking for documentation and clarification on the process for each application and what to expect. Your mileage may vary, don’t take my word for it, always have a good backup, and certainly don’t blow your voice servers up. Check the docs, check with TAC. Note that for each application, the changes are made on the publisher/primary server first, then any subscribers or secondary servers.

For CUCM and UNCX servers:

-In the GUI of the License server, remove the CUCM or UNCX server instance from the License server.  Yes, I know the trepidation of deleting anything from a voice server – especially involving licensing, but it is strongly recommended to do so. If you forget or don’t do this, PROBABLY nothing will happen, according to my conversations with TAC.  But no way I’m taking that chance, you decide for yourself…

-At the CLI of the server, issue the following commands:

set network dns primary X.X.X.X
set network dns secondary X.X.X.X
show network eth0 – confirm the changes

-Add the instance back to the license server and synchronize. When adding the server back, remember it’s the OS username and password that you want to be using.

-Additional step for CUCM highly recommended by TAC: restart the Cisco Tomcat service at the CLI with the command utils service restart Cisco Tomcat

-This is also the process for stand alone license servers, but of course you don’t have to remove any instances from the license server or perform the Tomcat restart.

For IM&P servers:

-At the CLI of the server, issue the following commands:

set network dns primary X.X.X.X note you will still get an error message that rehosting is required, but I confirmed later with TAC that this is just a holdover error message from the 8.x days.
set network dns secondary X.X.X.X
show network eth0 – confirm the changes

For UCCX servers (note I have an HA environment):

-Take note of the current server license MAC, put it in a safe place.  I copied the contents of the license files to my desktop and took a screenshot of the current license configuration page. Because it’s voice and the paranoia with licensing runs deep.

-Sacrifice a chicken or two, and then at the CLI of the server, issue the following commands on each server:

set network dns primary X.X.X.X
set network dns secondary X.X.X.X
show network eth0 (confirm the changes)

-Reboot the primary, wait for what feels like an eternity for the primary to come back up and get it’s services started, then reboot the secondary.

-Take note of the new license mac and request a rehost, provide licensing with the new and the old license macs.

-Load the new license file and start happy dance.  Unless you hit an issue like I did and the new license won’t load. Then try another reboot of the pair, attempt license load once more with fingers more tightly crossed than before, and then proceed to happy dance.

Last, but not least, my good twitter friend and awesome voice guru Ryan Huff pointed out to me this Answer File Generator tool which can be used to predict your new license mac so you can request a rehost in advance. I decided that the 30 day grace period for UCCX would be enough for this project, but it’s fantastic to know that such a thing exists. Especially if you are going to be invalidating a lot of licenses, have a very small change window, and want to go ahead and get off grace period licensing as quickly as possible*.


*my rehosted license file for UCCX was generated in under 15 mins. Impressive. Still a PITA, but at least a quick PITA…

Published 09/09/2015










Tags: , , , , , , , , , ,