RSS

Forward Networks – go ahead, break it.

When you’re tasked with planning for data center failover testing, you spend an awful lot of time reviewing configurations and scenarios, scrutinizing every detail to ensure that when the plug is pulled – both figuratively, and in some cases, literally, that all will go according to plan.  If you are someone lucky enough to have a lab environment at your job, it’s usually only a partial reconstruction of the network at best. In many cases, the luxury of a lab is simply non-existent in the workplace. I tend to exist in that latter world…

Watching Forward Networks present at Network Field Day 13, I couldn’t help but think how great this solution would be for exactly these types of scenarios.  Sure, you can plow through configurations manually and predict with some certainty that your routing is resilient. However, what if you could run through failover scenarios and network changes in advance, actually see the impacts in a lab that faithfully reconstructed your entire network?  The confidence in the DR testing plan skyrockets, and the reliance on anti-anxiety meds and lucky rabbit feet plummets.

The Forward Networks solution allows you to do just that by basically pulling all your configurations from your production gear, building your network, and then letting you break it. You could also just evaluate the network as well, if you’re not feeling particularly destructive. Forward Networks has several built in checks for elements that are commonly misconfigured, such as port channels, vlans, and port duplex settings, pretty much letting the lab network point out your previously overlooked mistakes.

You can also use Forward Networks to determine the complete path of certain traffic using their rather snazzy UI, which allows for some intuitive queries formed in human-speak, not SQL-I-don’t-know-the-right-table-name-please-just-show-me-my-data format.

Forward looking at the Forward Networks solution (see what I did there?) – I do wonder if price will be an obstacle for small to medium enterprise, as several products in this space are reassuringly expensive.*

I love that there is already a long list of vendors whose gear is supported in the product, but keeping pace with new vendors and OS versions will be a certainly be a challenge – one Forward Networks sounds excited to take on.

Definitely check out David Varnum’s post on Forward Networks as well, he goes into some detail on the company, the APIs of the product, and configuration checks Forward Networks is capable of in it’s current release. He’s also included some nice screen shots of the UI.

All of the videos from NFD13 from Forward Networks are a good watch, but if you only pick one, don’t miss the simulated outage demo.  You’ll laugh, you’ll cry, you’ll be totally impressed by how much fun watching a pretend network failure can be.

 

 

*reassuringly expensive: a term I credit to the one and only Greg Ferro and a term that I make frequent use of in networking.

Published:

Disclaimer: While Networking Field Day, which is sponsored by the companies that present, was very generous to invite me to this fantastic event and I am very grateful for it, my opinions are totally my own, as all redheads are far too stubborn to have it any other way.

 

 

Tags: , , , , , , , ,

Bye, bye 2800s…

At the end of this month, the long beloved 2800 series voice gateways go end of support.  If you find yourself finally getting around to replacing these boxes with the 4000 series voice gateways, you’ll be happy to hear that transitioning PRIs from one to the other is relatively painless for voice work*.

I found it did take me a few minutes to hone in on the shiny new syntax for the clocking commands, so here’s a quick overview of what you’ll need to know.

We’re all pretty used to the standard “network clock participate, blah, blah, blah…” command paired with the “network clock select something-or-other…” command to keep those slip seconds on the T1 controller away.**

With the 4Ks, you need something similar, but of course, they couldn’t just leave the syntax the same. Instead you need to use something like this, of course using the slot numbers that your T1 card is installed in:

network-clock synchronization automatic
no network-clock synchronization participate 0/1

a previous version of this post had these commands:

network-clock input-source 1 controller t1 0/1/0
network-clock sync participate 0/1

but a very smart TAC engineer alerted me to bug ID CSCvb01800 which has to do with how the NIM is included, or rather, not included in the clocking. This changes the configuration in two important ways – one, forget the input-source command. Secondly, enter the no network-clock synchronization participate 0/1 command even though you don’t see network-clock synchronization participate 0/1 in the configuration.  This command is the default and not visible, even in the show run all.  If you followed my previous version of this post’s commands, you simply need to do a no network-clock input-source 1 controller t1 0/1/0 and a no network-clock sync participate 0/1.

And don’t forget to add “clock source line primary” on the controller port – you didn’t typically need to explicitly set this on the 2800s/2900s, but apparently the 4Ks need more hand holding and direct instructions.

controller T1 0/1/0
 framing esf
 clock source line primary
 linecode b8zs
 cablelength long 0db
 pri-group timeslots 1-24

When you get this right, you should see some good news similar to this scroll across the screen – feel free to do a little happy dance:

*Sep 9 21:11:01.139: %NETCLK-6-SRC_ADD: Synchronization source T1 0/1/0 is added to T0 selection process.

And you can check your T1 clocking information when you bring the circuit up with this handy little command:

#show network-clock sync
Symbols:     En – Enable, Dis – Disable, Adis – Admin Disable
             NA – Not Applicable
             *  – Synchronization source selected
             #  – Synchronization source force selected
             &  – Synchronization source manually switched

Automatic selection process : Enable
Equipment Clock : 2048 (EEC-Option1)
Clock Mode : QL-Disable
ESMC : Disabled
SSM Option : 1
T0 : T1 0/1/0
Hold-off (global) : 300 ms
Wait-to-restore (global) : 300 sec
Tsm Delay : 180 ms
Revertive : No

Nominated Interfaces

 Interface            SigType     Mode/QL      Prio  QL_IN  ESMC Tx  ESMC Rx
 Internal             NA          NA/Dis       251   QL-SEC    NA        NA       
*T1 0/1/0             NA          NA/Dis       1     QL-SEC    NA        NA

My apologies for a messy looking post, but due to the bug ID mentioned before, the verifications I mentioned in the previous version of this post change. For now, I am leaving the prior text in with strike-through, so anyone who read the previous post can see the changes.  

Using the commands adjusted for the bug ID, you see this instead:

#show network-clock sync
Symbols: En – Enable, Dis – Disable, Adis – Admin Disable
NA – Not Applicable
* – Synchronization source selected
# – Synchronization source force selected
& – Synchronization source manually switched

Automatic selection process : Enable
Equipment Clock : 2048 (EEC-Option1)
Clock Mode : QL-Disable
ESMC : Disabled
SSM Option : 1
T0 : Internal
Hold-off (global) : 300 ms
Wait-to-restore (global) : 300 sec
Tsm Delay : 180 ms
Revertive : No

Nominated Interfaces

Interface SigType Mode/QL Prio QL_IN ESMC Tx ESMC Rx
*Internal NA NA/Dis 251 QL-SEC NA NA

After cutting over to the new gateway, you can check for slip seconds on the line using the old standard “show controller T1 0/1/0” – but be sure you clear the counters after plugging in the circuit, since there are always a few slip seconds reported when first plugging in the circuit.

After resetting the counters, check that the slips stay at zero.

#show controller t1 0/1/0
T1 0/1/0 is up.
  Applique type is Channelized T1
  Cablelength is long gain36 0db
  No alarms detected.
  alarm-trigger is not set
  Soaking time: 3, Clearance time: 10
  AIS State:Clear  LOS State:Clear  LOF State:Clear
  Framing is ESF, Line Code is B8ZS, Clock Source is Line Primary.
  Data in current interval (566 seconds elapsed):
     0 Line Code Violations, 0 Path Code Violations
     0 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins
     0 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs

[output truncated]

Bonus material – the other strange syntax issue I hit with the 4Ks was bug ID CSCup86596 and the command “isdn incoming-voice voice” command wouldn’t take under the signaling channel configuration – the workaround is epic in its vagueness, noting that “the functionality is there and will work” even though you cannot enter the command.  So, yeah, typical voice shenanigans…

More bonus material – I was made aware by the previously mentioned very smart TAC engineer, that if you are looking into RTP-NTE (RFC 2833) DTMF issues on a 4K, you are going to need packet captures, the debug voip rtp sess name doesn’t do the trick.  I haven’t had to face this one yet, but hopefully that little bit of information will save you some time if you do.

 

*voice is pain, any one who tells you otherwise is trying to sell you something  😉

**to set clocking for PRIs on 2800s/2900s, typically you use something like these two commands, depending on which slot the WIC card is plugged into. If you are seeing slip errors, you should check for these commands: 

network-clock-participate wic 1
network-clock-select 1 T1 0/1/0

Published 10/21/2016

 

 

 

 

Tags: , , , , , , , , , , ,

Upgrading UCCX 9.0(2) to 10.6(1)

If you’re wondering about upgrading UCCX from 9.0(2)SU2 to 10.6(1)SU2, and would like that information with a side of snark, then this is the post for you.

Fair warning, this is one of those your-mileage-may-vary entries, definitely do not take my notes as gospel because I promise *I* won’t be the one restoring your UCCX server from backup at 3am.  Always read the release notes and upgrade guides in their entirety, skipping pages earns you nothing but (more) pain and suffering.

That being said, there’s a few important compatibility matrices you will want to check for this upgrade.

Pay close attention to whether your UCS can handle later versions of ESXi without UCS upgrades. For instance, 10.6 no longer supports ESXi 4.X, so you might have some pre-upgrade UCS work to be done before the real party gets started. If you are unsure about your UCS/ESXi version compatibility, try this link for checking.

UCCX 10.6 upgrade from 9.0(2) does require a .cop file, I used ciscouccx.refresh_upgrade_v1.9.cop.sgn.

Before beginning this process, I highly suggest taking a screenshot of your licensing, sacrificing a couple of chickens, and making sure you have ordered your upgrade license through the PUT tool. Voice engineering breeds much in the way of paranoia, so I also recommend downloading the scripts and prompts, just in case you get the additional fun of rebuilding the server from scratch. It should go without saying, but I’ll say it anyway, be sure to check that your backups have been running AND that they have been running successfully.

The order of operations goes something like this for an HA setup, note that there is no Finesse deployed in my setup, strictly CAD.

  • Confirm the primary is active and all services show IN SERVICE – don’t skip this, I’ve never tried to upgrade in an active failed over state, but I imagine it’s like crossing the streams and would end in much badness.
  • Install .cop file on the primary, reboot, grab coffee, wait for services to come back up
  • Install .cop file on the secondary, reboot, grab moar coffee, wait for services to come back up
  • Install upgrade file to primary, drive to a different county to get coffee, don’t panic when the server reboots during the installation, and do not reboot after install.
  • Install upgrade file to secondary, switch to vodka. Question life choices to get involved in voice engineering.  Do not reboot after install.
  • Switch versions on the primary. There’s more than enough time at this stage to continue questioning your life choices. All of them.
  • Server (finally) boots to new version. Wait for services to start. The docs say this could take up to 30 minutes. Shouting profanities at the server will not shorten this interval significantly, but you’ll likely try anyway.
  • Log into the server, install license file, note the error message about OVA template issues. Shut down the server because seriously, who needs that kind of negativity in life? Or shutdown because you need to make changes to the OVA. Whichever…
  • Modify OVA template for RAM, OS, and vNIC changes*
  • Power on server, wait for services. Yes, again.
  • Switch versions on the secondary, repeat the process above, pour another glass of whatever is left.
  • Once the primary and secondary are both online with all services show IN SERVICE, check that replication status is good. 
  • Run the client configuration tool and test your queues.  Buy a lottery ticket if you haven’t had to call TAC,by this point.**

The above list is strictly an overview, but gives you a reasonable idea of what to expect during the upgrade. A whole lot of proper planning will result is a whole lot of waiting for things to happen, but not much else.  An uneventful voice upgrade is an awesome voice upgrade.

 

*Things got interesting with this. During the planning stages, TAC sent me a link to the procedure for altering the OVA, related to Bug ID CSCut04158. This detailed the process to change the vNIC to vmxnet3. When I presented the process to my virtualization guru, he concluded it would not work for our configuration, we would have to use some PowerCLI magic instead. And by we, I mean he. He used PowerCLI magic, I just threw another chicken onto the alter. The code went something like get-vm MyServerName | get-networkadapter | set-networkadapter -type “vmxnet3” – use this suggestion at your own risk, I am not a virtualization expert, nor do I play one on TV. 

**I will point out that in the upgrade guide, an error about unaligned partitions is called out as a potential issue – it sounded like a whole lot of no fun resulting in rebuilding from backup, and I was quite relieved I didn’t hit that one. Did I mention read the docs?  Definitely do that…

Published 08/16/2016

 

 

 

 

 

 

 

 

 

 
6 Comments

Posted by on 2016/08/16 in UCCX, Upgrades, Upgrades

 

Tags: , , , , ,

2016 Cisco Live, US – Geeks for life.

Putting together a wrap up post on Cisco Live US always makes me smile, and 2016 is no exception.  As many of you know, this CLUS marked 5 years since Tom and a small group of engineers first bonded over networking nerdiness and an addiction to 140 characters.

We’ve followed and helped each other through upgrades, outages, career changes, certifications, and a plethora of engineering challenges. We’ve commiserated with the suck that sometimes is our jobs, we’ve championed the hard-fought successes of our peers, and as a bonus, we’ve managed to provide an ever flowing stream of hilarious commentary along the way.

I’m thrilled to be part of a networking community that just keeps growing larger, and I’m continually impressed with the engineering talent this group represents.  I’m also pleased to have been part of Tech Field Day at CLUS as well, which continues to do an amazing job bringing great content and engineers together.

Just check out this awesomeness. Could there be anything better? No, no there could not.

 

Tags: ,

8841s and line selection woes…

Just a quick post in case there is at least one other person in the world that didn’t know this particular trick for the 8800* series phones that always go off hook on the last line used.  In case you weren’t aware that these phones had this behavior, don’t worry, as soon as you deploy one to a receptionist that handles more than a single phone line, you WILL be made aware. You may not survive the encounter, but you will know…

Should the user let you live long enough to research a solution, you will find that TAC doesn’t offer you any hope in restoring the requisite amount of receptionist happiness to keep oxygen flowing to your lungs. TAC will politely inform you that the phone is operating as designed and that you might want to buy the receptionist some flowers instead.  Okay, maybe that last part was my idea…

Technically TAC is right, you cannot force the phone back to selecting the primary line after hanging up a call that was answered or placed on any of the other lines. However, as my awesomely brilliant coworker discovered, you can change the behavior of the phone so that when calls come in they are DISPLAYED on the primary line.

This way, when the user hangs ups, the phone still has the primary line selected, even if the last call received was for one of the other lines on the phone. This mimics a behavior users have come to know and love of the older generation of phones, a behavior users who manage multiple lines desperately want back.

You can set this option on the phone configuration page, just search for Show All Calls and change the feature to Enabled.

Show All Calls on Primary Line

This won’t make the remembering-the-last-line-selected-for-off-hook behavior go away, but it does make its impact far less noticeable to end users who might otherwise want to plot your demise.

* In this case we were working with 8841 phones, but my coworker has used this trick in the past on 9971 phones, so I’d be inclined to believe this would work on pretty much any SIP phone that has this behavior. 

Published 04/11/2016

 

Tags: , , , , , ,

Tom’s Corner – Cisco Live Twitter community 5 years later…

If you’re an annual attender of Cisco Live like I am, you might be noticing there’s some pretty cool stuff taking shape for 2016.

This year will be the 5th anniversary of Tom’s Corner – the first ad hoc gathering location for us geeky, twittering engineers*.

Hard to believe, but in 2011 we were just a smallish group of network nerds, some with quite a bit more hair in those days, and of course, the one random photo-bomber guy on the left. Still nobody knows who that guy was…

The Twitter Group 2011

The amazing amount of talent represented and the camaraderie of the network engineering community is what makes Cisco Live the place to be.

This year I’m also looking forward to witnessing some epic Engineering Deathmatch action. If you aren’t familiar with Engineering Deathmatch you gotta watch this video.  You won’t be disappointed.

And if that weren’t enough Cisco Live goodness, I gotta admit I’m pretty excited about hearing Keyser Söze, err Kevin Spacey, speak.

Other useful details for attendees:

  • The scheduler will be available on  April 26th for NetVets, May 3rd for the rest of us.
  • The band this year is Maroon 5, and the CAE is going to be at the shiny new T Mobile Arena
  • Newbies to Cisco Live can get a super cool mentor to help them figure the conference thing out, just sign up here
  • Tech Field Day will be at Cisco Live this year as well, look for fabulous content from them. If you’re interested in being a delegate for this or any event, apply here
  • Awesome blogs to watch for new Cisco Live content and to read past posts for excellent conference advice: @fryguy_pa, @scottm32768, @mrtugs@net_introvert, @networkingnerd, @someclown, @aconaway, @danieldibswe, @gingmarCisco Live Attendee Blogs
  • The annual twitter list is out, sign up here!

*Tom’s corner even had it’s own check in on Four Square. In case there was any doubt we were and still are total nerds…

Published 04/11/2016

 
Leave a comment

Posted by on 2016/04/11 in Cisco Live

 

Tags: , ,

Cisco Live Europe: NBASE-T progress report

The Ether

Robert Metcalfe –May 22nd, 1973 – first memo describing Ethernet. Photo Credit: http://www.ieee802.org/3/ethernet_diag.html

If you ever get a chance to watch Peter Jones present on a topic, I highly encourage you to take it! He’s passionate about multigigabit and the evolution of Ethernet technology, and it shows.

In this CLEUR Tech Field Day Extra session, Peter outlines significant progress that has been made on the 2.5G and 5G standardization process, which is pretty impressive considering the NBASE-T Alliance was only formed in October of 2014.

In addition to announcing some Cisco products that will be capable of multigigabit magic, Peter also lays out some additional interesting use cases, besides just the hotly debated timeline of the impending 802.11ac Wave 2 data rate apocalypse.

A few highlights from the product announcement pieces of the presentation, be sure to consult relevant data sheets for accuracy, caveats, and any insomnia issues you might have:

  • 48 port blade for the 4500E, 12 ports will do multigigabit
  • 3850 24 or 48 port switch, half of the ports will do multigigabit; note this is an entirely new SKU but stacking is compatible with previous 3850s
  • 3560-CX compact switch, two ports will support the multigigabit
  • All multigigabit ports are supporting UPOE (60 Watts)
  • No SFP+ available – Peter talks briefly about the heat challenges around this form factor, interesting geeky stuff
  • 3800 series AP

While I agree that there’s certainly some questions as to which devices and their data rates will bring current copper infrastructure to its knees, realistically getting more mileage out of already installed cabling makes a good kind of sense. Hardware refresh cycles are generally much shorter intervals than building wiring replacement schedule. Finding ways to cope with aging copper cabling and eking out a few more years of use from such a significant investment is a reality for engineers.

To leverage 2.5G or 5G magic, both devices – what’s getting plugged in and what’s being plugged into – will need to support the multigigabit technology.  Coordinating equipment refresh cycles and budget to make this happen could be quite the challenge as well.  Some number crunching would be appropriate to determine cost effectiveness for your own environment, but multigigabit Ethernet could be that additional tool in the belt when it comes to maximizing cabling investments already made.

Also of nerdy note, Peter’s short discussion of how the multigigabit magic filters out the noise using advanced coding and signal processing is quite fascinating. These brief comments led to me down a rabbit hole and to this interesting article,Crosstalk problems are back, which discusses the issues of noise when dealing with high data rates across copper wiring.

I highly recommend checking out these posts from other awesome engineer bloggers, good stuff:

http://ipengineer.net/2015/02/multigigabit-ethernet-2-5-5-0gbps-nbase-t-alliance/

http://hansdeleenheer.com/cisco-nbase-t-lipstick-on-your-cat5e-pig/

http://sc-wifi.com/2015/03/03/the-evolution-that-will-start-the-revolution/

http://networkingnerd.net/2015/02/03/nbase-ing-your-wireless-decisions/

And for more learning on the topic, definitely check out these:

http://techfieldday.com/appearance/cisco-enterprise-presents-at-tech-field-day-extra-at-cisco-live-milan/

http://www.cisco.com/c/dam/en/us/solutions/collateral/enterprise-networks/catalyst-multigigabit-switching/multigigabit-ethernet-technology.pdf

http://www.cisco.com/c/en/us/solutions/enterprise-networks/catalyst-multigigabit-switching/index.html

http://www.cisco.com/c/m/en_us/training-events/events-webinars/webinars/techwise-tv/802-11ac-wave-2.html

http://www.cisco.com/c/en/us/support/interfaces-modules/catalyst-4500e-48-port-upoe-12-mgig-ports-rj45-line-card/model.html

http://www.networkworld.com/article/2840287/cisco-subnet/cisco-others-pushing-2-5g-5g-ethernet.html

http://www.networkcomputing.com/wireless/80211ac-wave-2-multi-gigabit-ethernet-push/1691916854

http://www.networkworld.com/article/2873116/cisco-subnet/scared-of-802-11ac-wave-2-wi-fi-here-comes-cisco-multigigabit-to-the-rescue.html

http://www.cisco.com/c/m/en_us/training-events/events-webinars/webinars/techwise-tv/802-11ac-wave-2.html

Fundamentals of NBASE-T

http://www.ccierants.com/2015/01/new-cisco-switches-40-gig-uplinks.html

http://techfieldday.com/appearance/cisco-enterprise-presents-at-tech-field-day-extra-at-cisco-live-milan/

http://www.theregister.co.uk/2015/12/10/nbaset_maps_out_spec_ahead_of_products_in_2016/

 

Standard Disclaimer: Tech Field Day covered my expenses at Cisco Live Europe, but I am a redhead, any thought that my opinions could be bought or dictated is just crazy talk. 

 
 

Tags: , , , , ,