Network Change Validation Meets Supersized Network Emulation

Large scale networks means large scale configuration and change management testing. Or at least it should.

But device expense, power costs, and space limitations mean full scale physical network labs don’t happen. We, the engineers, get to roll-out complex network changes based on limited tests and what we hope is a well thought out, bulletproof rollback plan. We often risk significant loss of revenue for the company and significant loss of sleep for ourselves if changes go poorly.

This is not just a big shops problem either.  Even – or perhaps especially, small to medium enterprises lack full scale physical labs to simulate changes.  I’ve known one engineer that used to have a Nexus 5K on his desk (I’m looking at you @that1guy_15), but most of us are lucky to have a few pieces of equipment to cobble together to give us the general gist of the impact of a potential network change.

With the cloud eating everything, it’s about time that it started giving back to engineers – and that is what Tesuto seeks to do.  Tesuto leverages cloud to perform large scale emulation of networks, while allowing engineers to leverage modern automation tools and testing along the way.

Tesuto spins up emulation devices in Google Cloud or Digital Ocean, with support coming soon for Azure, AWS, and private cloud as well.  These spun-up devices have full L2 connectivity with each other and are running the actual vendor images, giving engineers emulations that can accurate reflect control plane functionality for configuration and change testing at the scale your network demands.

It’s worth noting that if you want to test ASIC specific functionality or throughput testing, this is not the platform for those types of tests. Emulations are ideal for control plane and connectivity testing, such as making BGP routing changes and seeing what neighbor relationships you hosed up, but not so ideal for how many packets per second a device can spit out.

So why not use GNS3, which offers device emulation as well?  Resource scale, ease of use, Rapid Initialization*, and the ability to tie into modern automation configuration and testing tools such as Ansible, NAPALM, etc…, are just a few reasons why cloudifying your network emulations with Tesuto starts to make sense.

Personally, I found the interface to be pretty intuitive, creating a few routers from different vendors, connecting them, and logging into the Tesuto provided jumpbox was quick and painless. Uploading licensed images for some vendors is required, so be prepared to BYOI (bring your own image).

The ability to run built in NAPALM validation tests takes a bit more finesse and experience, as does integrating Tesuto into your automated change management pipeline if you have one. With a bit of additional work, though, you can create your own validation tests, you are not limited to the built in tests or to NAPALM.

Tesuto brings a ton of additional features to network emulation, as you can see from the chart below. I recommend watching both NFD21 presentations, especially this demo in which a lot of questions you don’t even know you have yet are answered.

*Tesutos’s Rapid Initialization is a feature which significantly decreases boot time of the devices after first power on, so that MX router that takes 25 mins to boot the first time in an emulation takes only 5 mins on future boots. 

 

Disclaimer: While Networking Field Day, which is sponsored by the companies that present, was very generous to invite me to the fantastic NFD21 event and I am very grateful for it, my opinions are totally my own, as all redheads are far too stubborn to have it any other way.

Published 10/13/2019

Cisco Live 2019 – A Whirlwind of Networking Goodness

Cisco Live 2019 came and went in a whirlwind of fantastic meetups, excellent sessions, and genuinely nerdy networking conversations.

Cisco Live session content was top notch. Jasper Bongertz’s Wireshark talk blew the audience away with useful packet capture and troubleshooting tips, and Denise Fishburne’s Network Detective presentation captivated the audience with methodical troubleshooting processes and issue isolation techniques. Both sessions are a must watch for network engineers. Seriously, you will thank me (send coffee) and more importantly you should definitely thank them for giving so much to the community!

Tech Field Day captured a ton of great content this year as well.  I especially recommend this NetBeez presentation highlighting the exciting ways their monitoring solution is fighting the good fight by helping to prove it’s not the network. Their new integration with Cat9K switches is also covered and definitely worth checking out.

This year also featured the distribution of Amy’s Army of Angry Routers. Angry routers were given, angry routers were received, and a new site header came to be.

Cisco Live 2019 was also especially memorable in the recognition that this very blog received! As Cisco 2018 IT Blog Award winner for Most Entertaining, yours truly had her big screen moment! I couldn’t be more thankful for each of you who took the time to vote! Thanks for reading along, laughing along, and sharing along with my adventures, snark, and bits of wisdom. You all rock, and obviously have the best taste.

 

And finally, my favorite part of every Cisco Live wrap up, the photo gallery! So many long time friends, so many new friends.  The networking community is genuinely the best and you all make it that way.

Published 06/23/2019

“Thirteen hundred APs, no open support tickets” – achieving quality in wireless networks

“Thirteen hundred APs, no open support tickets,” Sudheer Matta, VP of Products for Mist Systems, boldly stated during his MFD3 presentation.  At the time, he was referencing one of their largest customers specifically, but the company’s desire to prevent bugs, create high quality customer experiences, and resolve issues quickly were principles that permeated the discussions with Mobility Field Day delegates.

Mist leverages several key components in order to pull off their customer focused reliability, visibility, and proactive troubleshooting of the wireless network.

Cloud-based micro-services architecture.  This modern approach to building systems is part and parcel of what many cloud companies have been doing with their software architecture over the last few years.  Instantiating distributed services and leveraging APIs between these services is foundational to providing the kind of resiliency and redundancy cloud makes possible and Mist credits this architecture with how they are able to push out new features, fixes, and services weekly without causing any data plane outages for customers.

In his presentation, Sudheer shares an impressive case of how Mist was able to do a complete restore for a customer that had deleted their entire controller infrastructure. All the controllers and services were back online in less than 2 hrs with no access point reboots or data plane outages, a feat Sudheer also credits to Mist’s distributed architectural approach.

Analytics. These days collecting data is table stakes, the real advancement is in building better algorithms that provide useful information to customers. Mist calls these, “actionable insights” and they are more than just increasing the noise floor with more alerts. Mist believes their actionable insights are so dead on that they’ve announced proactive anomaly detection, meaning the system will open a ticket on your behalf when an issue is detected.

And the analytics don’t stop with just ticket opening – MARVIS (Mist’s AI) is getting several feature enhancements focused on improving the troubleshooting process, reducing analysis time, and improving RRM.

A culture of attention to detail. After watching Mist’s MDF3 presentations, I would describe their business model as “just good enough is not good enough for us.”

Besides a distributed architecture designed to minimized the number of bugs and the impact of those bugs that do make it into the system, issues are expected to be resolved quickly and not allowed to fester or be ignored.  A clear emphasis is placed on quality and usability of the system, from the architecture to the user experience.

Mist is also listening both to its customer base as well as wireless engineers. An improved adapter bracket, the transparency with firmware version issues, the coming soon red and green buttons, and the constant tuning of the virtual assistant were just a few indicators from the presentations that customer experience and usability not only matter, but are at the top of the priority list.

For more Mist goodness, be sure to check out these posts:

@badger-fi – Mom’s love Mist

@rowelldionicio – Demistifying Wi-Fi Issues

@Drew_CM – Mist Enhances Machine Learning Capabilities To Improve WLAN Performance, Troubleshooting

@theitrebel – MDF Day 1 Recap

Disclaimer: While Mobility Field Day, which is sponsored by the companies that present, was very generous to invite me to the fantastic MDF3 event and I am very grateful for it, my opinions are totally my own, as all redheads are far too stubborn to have it any other way.

Published 10/7/2018

 

Intro into Fortinet WLAN configuration

Simple, secure, sensible – Koroush Saraf, Fortiner VP of Product Management, emphasized these words in his recent MFD3 presentations. While any vendor can claim their products share these attributes, it’s usually the complexity of workflow that reveals the betrayal of one or all of these characteristics. Watching this Mobility Field Day demo, however, the simplicity of setting up a basic Fortinet WLAN SSID, applying security policies, and even setting up automation for quarantining an infected machine boiled down to just a few steps.

Step oneCreate your SSID.

In Fortinet world, creating an SSID creates a virtual interface.  At first, this seems like a strange construct to be involved in a WLAN setup process, but later in the process, the logic and flexibility of having this virtual interface becomes apparent.

To create your basic SSID, navigate to WiFi & Switch Controller, click on SSID, click Add New.  You can select if this SSID will be a Tunnel, Bridge, or Mesh SSID, as well as configure parameters such as IP address, DHCP server options, Default Gateway, DNS servers, etc…

Keep in mind that to avoid clutter, the GUI presents the essential and the most commonly used options for configuration. Some more advanced configuration may not be seen in the GUI but available via CLI.

Step 2: Attach or create an AP Profile.

The FortiAP Profile is where things like radio bands, transmit power, channel and channel width, etc… are configured and controlled in a manner that can be applied to multiple APs.

To create a new AP profile, navigate to WiFi & Switch Controller, and click on FortiAP Profiles, click Add New

To attach an already created AP Profile to an AP, navigate to WiFi & Switch Controller, click Managed FortiAPs, select your AP, and apply the appropriate profile to the AP. This screen is also where you would configure AP specific options that would not apply to all APs using the profile selected. Note, this assumes you have already setup your basic controller parameters so that APs can be automatically discovered.  For more information, see the documentation cited at the end of this post.

Step 3: Create interface policies.

This step brings together the SSID virtual interface created and the security policies that need to be applied to the SSID.  The virtual interface allows for the straight-forward application of security policies such as allowed/denied ports and protocols, along with UTM features and application restrictions.

For engineers that have configured Fortigate firewalls, this part of the process will feel the most familiar since it’s leverages the same process of policy creation used to create traditional firewall rules. 

Bonus step: Configuring an automation alert for compromised clients.

Now that you have your SSID and AP online, you can head over to Automation and quickly setup workflow for what should happen when the Fortigate sees a compromised host. You can see from the screen shot below that not only can the host be quarantined automagically, but an email could be sent to inform those taking the calls from the angry virus-spreading-machine owners that these machines have been blocked.Note this type of automation can apply not just to WLAN clients, it is a feature that can be used globally for all detected endpoints.

To see this demo in action, check out this MFD presentation in which Fortinet makes a compelling case for the idea that the lives of IT engineers shouldn’t be made so difficult all the time. Now if only all IT vendors thought this way…

And for even more Forti-content, check out these posts from fellow delegates:

Lee BadmanClarity and Confusion- Fortinet and Arista at Mobility Field Day 3

Scott LesterForti What

Jim PalmerA Story of Three Companies

 

 

Note: This post is based on the basic setup and topolgy given in the video presentation, for more advanced configuration information, please check out Fortinet’s documentation that can be found here. Also, Fortinet has an pretty awesome demo site here which allows you to log in and look around in pretty much any Fortinet product you’d like to see.

Disclaimer: While Mobility Field Day, which is sponsored by the companies that present, was very generous to invite me to the fantastic MDF3 event and I am very grateful for it, my opinions are totally my own, as all redheads are far too stubborn to have it any other way.

Published 9/30/2018

Runt Post: Quality troubleshooting, what it looks like

In my previous post, I shared some of the cool stuff ThousandEyes is doing with VoIP.  I also wanted to draw attention to this cool video of Mohit Lad, co-founder and CTO of ThousandEyes, using his own product to troubleshoot an outage event on the fly: http://vimeo.com/105805525

There are very few ways to show off your product better than this type of demonstration. Mohit troubleshoots with expertise, clearly in his element. The tools cater well to his methodical troubleshooting process and both are quite impressive. Plus the routing loop he finds is just darn cool.

photo 100000 (8)

Watch it, you’ll love watching a master at work, I know I did.

Published: 9/26/2014

Disclaimer: While Networking Field Day, which is sponsored by the companies that present, was very generous to invite me to this fantastic event and I am very grateful for it, my opinions are totally my own, as all redheads are far too stubborn to have it any other way.

 

Voice Girl Goes to Storage Day

Who has two thumbs and got to attend the last Tech Field Day?  This girl!

In case you don’t know what Tech Field Day is, go here and check it out:  http://techfieldday.com/   In case you don’t care what Tech Field Day is, I suggest you stop reading or make sure you have copious amounts of alcohol handy.  Actually, that last suggestion could improve the reading of any of my posts, so feel free to get started, you have my blessing.

Now, I’m sure we’ve all had that friend who goes on a vacation and brings back 10,000 pictures and insists on narrating them all in great, painstaking detail.  Fear not – I want to smack that guy as much as you do – so I will just be hitting the highlights of this expedition in this post.

So, without further ado, awesome thing number 1: hanging out with server admins.  I know, I know, for network and/or voice guys this hardly sounds like something that would make the list of awesome- unless that list were titled Ways In Which My Day Could Awesomely Suck – but it’s true and let me tell you why.

With roles in IT becoming less and less siloed, it’s clear us folks guarding the layer 2 and 3 keys to the castle are going to have to make nice with those folks rocking the upper layer data center knowledge.  As distasteful as that may initially sound to both parties involved, we all earn huge rewards.

Think about it- do you really want that server guy vMotioning all those production boxes across your precious WAN without any clue as to the implications?  I’m certain that server guy with the ponytail doesn’t want us well-intentioned network junkies screwing with SAN infrastructure when he/she thinks we don’t even know what random IO is. Of course do we do know what it is, but not the point…

Contrary to popular sysadmin belief, we network folks are capable of reading and do in fact know what a manual looks like.  Contrary to network admin belief, server guys do know what they are doing and don’t just break crap on purpose.  Given shrinking IT budgets, device consolidation, and technology overlap, our tiny sandbox has only gotten tinier and now it looks like we’re going to have to share the dump truck and not just the buckets.  (the dump truck was always my favorite)

So awesome thing number two:  presentations! Companies solving problems I was vaguely aware existed in ways I only wish I had imaged because retirement would be nice about now.  The quality of presentations was generally high and the technical level generally deep.  Perfect combination.

Let me offer a few brief take-aways from what I saw, you can catch the presentations here http://vimeo.com/groups/techfieldday:

  • Nutanix: Putting your VMs and storage on the same devices, have them utilize the same resources.  It has a kind of eggs in one basket feel – but the basket is really nice.  Interesting implications on the necessity for SAN administrator. http://www.nutanix.com/
  • Nasuni: If you ever want tips on how to deliver a presentation, watch this one. The send-your-files-in-the-cloud-and-see-them-at-your-other-sites product was wicked cool. Matt Simmons had the product up and running during the time of the demo. Sweet. http://www.nasuni.com/
  • Symantec Storage Foundation 6.0: Least favorite presentation style. So. many. power. point. slides. Clearly this product has some significant improvements over the previous version but the demo certainly wasn’t showing off this products nice curves, so to speak. http://www.symantec.com/business/storage-foundation
  • Data Direct Web Object Scaler: large-scale cloud storage wow-ness.  Keeping track of your massive amounts of cloud data using custom filing system to store and replicate data. Demo was super neat, product super fast.   http://www.ddn.com/products/web-object-scaler-wos
  • Pure Storage- all SSD storage, forget tiering.  They wrote their own software to talk/write to SSD drives in a way that makes SSD drives very happy. In fact, drives never fail for Pure Storage, or so was said- a concept our little group of skeptics had some trouble with. Pure Storage held to their guns though and a promise was made to tweet the first drive failure. http://www.purestorage.com/
  • Arista EOS:  Command line goodness. In the demo, the guy added the XMPP package to the Linux-based software running the switch, then chatted with the switch. Totally cool. Who doesn’t want to ask a switch how it’s day is going? http://www.aristanetworks.com/
  • SolidFire- All SSD storage, optimized for providers who want to limit compute and/or storage on a per customer basis. If you are a cloud provider of storage, being able to establish very specific SLAs for customers I’m sure is extremely appealing.  http://solidfire.com/
  • Arekia- backup goodness.  Presentation went into detail on their particular brand of deduplication which provides quite a lot of benefit when backing up large amounts of data. http://www.arkeia.com/

Last but not least, awesome thing number three: Stephen Foskett and Matt Simmons are freaking fantastic!  As the organizers, they coordinated every intricate detail and then made it look easy to the rest of us.  A very special thanks to those guys for making all of this happen, wishing them happy times in therapy as they attempt to recover.

For links to all things Tech Field Day 8: http://techfieldday.com/2011/tfd8/