RSS

“Thirteen hundred APs, no open support tickets” – achieving quality in wireless networks

“Thirteen hundred APs, no open support tickets,” Sudheer Matta, VP of Products for Mist Systems, boldly stated during his MFD3 presentation.  At the time, he was referencing one of their largest customers specifically, but the company’s desire to prevent bugs, create high quality customer experiences, and resolve issues quickly were principles that permeated the discussions with Mobility Field Day delegates.

Mist leverages several key components in order to pull off their customer focused reliability, visibility, and proactive troubleshooting of the wireless network.

Cloud-based micro-services architecture.  This modern approach to building systems is part and parcel of what many cloud companies have been doing with their software architecture over the last few years.  Instantiating distributed services and leveraging APIs between these services is foundational to providing the kind of resiliency and redundancy cloud makes possible and Mist credits this architecture with how they are able to push out new features, fixes, and services weekly without causing any data plane outages for customers.

In his presentation, Sudheer shares an impressive case of how Mist was able to do a complete restore for a customer that had deleted their entire controller infrastructure. All the controllers and services were back online in less than 2 hrs with no access point reboots or data plane outages, a feat Sudheer also credits to Mist’s distributed architectural approach.

Analytics. These days collecting data is table stakes, the real advancement is in building better algorithms that provide useful information to customers. Mist calls these, “actionable insights” and they are more than just increasing the noise floor with more alerts. Mist believes their actionable insights are so dead on that they’ve announced proactive anomaly detection, meaning the system will open a ticket on your behalf when an issue is detected.

And the analytics don’t stop with just ticket opening – MARVIS (Mist’s AI) is getting several feature enhancements focused on improving the troubleshooting process, reducing analysis time, and improving RRM.

A culture of attention to detail. After watching Mist’s MDF3 presentations, I would describe their business model as “just good enough is not good enough for us.”

Besides a distributed architecture designed to minimized the number of bugs and the impact of those bugs that do make it into the system, issues are expected to be resolved quickly and not allowed to fester or be ignored.  A clear emphasis is placed on quality and usability of the system, from the architecture to the user experience.

Mist is also listening both to its customer base as well as wireless engineers. An improved adapter bracket, the transparency with firmware version issues, the coming soon red and green buttons, and the constant tuning of the virtual assistant were just a few indicators from the presentations that customer experience and usability not only matter, but are at the top of the priority list.

For more Mist goodness, be sure to check out these posts:

@badger-fi – Mom’s love Mist

@rowelldionicio – Demistifying Wi-Fi Issues

@Drew_CM – Mist Enhances Machine Learning Capabilities To Improve WLAN Performance, Troubleshooting

@theitrebel – MDF Day 1 Recap

Disclaimer: While Mobility Field Day, which is sponsored by the companies that present, was very generous to invite me to the fantastic MDF3 event and I am very grateful for it, my opinions are totally my own, as all redheads are far too stubborn to have it any other way.

Published 10/7/2018

 

Advertisements
 
1 Comment

Posted by on 2018/10/07 in Tech Field Day, Wireless

 

Tags: , , , , , ,

Intro into Fortinet WLAN configuration

Simple, secure, sensible – Koroush Saraf, Fortiner VP of Product Management, emphasized these words in his recent MFD3 presentations. While any vendor can claim their products share these attributes, it’s usually the complexity of workflow that reveals the betrayal of one or all of these characteristics. Watching this Mobility Field Day demo, however, the simplicity of setting up a basic Fortinet WLAN SSID, applying security policies, and even setting up automation for quarantining an infected machine boiled down to just a few steps.

Step oneCreate your SSID.

In Fortinet world, creating an SSID creates a virtual interface.  At first, this seems like a strange construct to be involved in a WLAN setup process, but later in the process, the logic and flexibility of having this virtual interface becomes apparent.

To create your basic SSID, navigate to WiFi & Switch Controller, click on SSID, click Add New.  You can select if this SSID will be a Tunnel, Bridge, or Mesh SSID, as well as configure parameters such as IP address, DHCP server options, Default Gateway, DNS servers, etc…

Keep in mind that to avoid clutter, the GUI presents the essential and the most commonly used options for configuration. Some more advanced configuration may not be seen in the GUI but available via CLI.

Step 2: Attach or create an AP Profile.

The FortiAP Profile is where things like radio bands, transmit power, channel and channel width, etc… are configured and controlled in a manner that can be applied to multiple APs.

To create a new AP profile, navigate to WiFi & Switch Controller, and click on FortiAP Profiles, click Add New

To attach an already created AP Profile to an AP, navigate to WiFi & Switch Controller, click Managed FortiAPs, select your AP, and apply the appropriate profile to the AP. This screen is also where you would configure AP specific options that would not apply to all APs using the profile selected. Note, this assumes you have already setup your basic controller parameters so that APs can be automatically discovered.  For more information, see the documentation cited at the end of this post.

Step 3: Create interface policies.

This step brings together the SSID virtual interface created and the security policies that need to be applied to the SSID.  The virtual interface allows for the straight-forward application of security policies such as allowed/denied ports and protocols, along with UTM features and application restrictions.

For engineers that have configured Fortigate firewalls, this part of the process will feel the most familiar since it’s leverages the same process of policy creation used to create traditional firewall rules. 

Bonus step: Configuring an automation alert for compromised clients.

Now that you have your SSID and AP online, you can head over to Automation and quickly setup workflow for what should happen when the Fortigate sees a compromised host. You can see from the screen shot below that not only can the host be quarantined automagically, but an email could be sent to inform those taking the calls from the angry virus-spreading-machine owners that these machines have been blocked.Note this type of automation can apply not just to WLAN clients, it is a feature that can be used globally for all detected endpoints.

To see this demo in action, check out this MFD presentation in which Fortinet makes a compelling case for the idea that the lives of IT engineers shouldn’t be made so difficult all the time. Now if only all IT vendors thought this way…

And for even more Forti-content, check out these posts from fellow delegates:

Lee BadmanClarity and Confusion- Fortinet and Arista at Mobility Field Day 3

Scott LesterForti What

Jim PalmerA Story of Three Companies

 

 

Note: This post is based on the basic setup and topolgy given in the video presentation, for more advanced configuration information, please check out Fortinet’s documentation that can be found here. Also, Fortinet has an pretty awesome demo site here which allows you to log in and look around in pretty much any Fortinet product you’d like to see.

Disclaimer: While Mobility Field Day, which is sponsored by the companies that present, was very generous to invite me to the fantastic MDF3 event and I am very grateful for it, my opinions are totally my own, as all redheads are far too stubborn to have it any other way.

Published 9/30/2018

 
1 Comment

Posted by on 2018/09/30 in Tech Field Day, Wireless

 

Tags: , , , , , , , , ,

Solving 802.11ad challenges

The 802.11ad (60GHz) market has been interesting to watch, especially now that products utilizing the technology are becoming more prevalent.  Vendors leveraging 802.11 in a frequency band whose propagation properties differ significantly from the traditional 802.11 bands of  2.4 and 5GHz mean new engineering challenges to overcome.

The AP-387 point to point unit announced by Aruba earlier this year seeks to address these challenges in a few interesting ways.  Rain fade is primary concern when working with high frequencies such as 60GHz.  In order to combat this, the AP-387 has two radios, a 60GHz and a 5GHz radio, with the unit aggregating the throughput of both radios.

Should a storm roll through, the unaffected 5GHz radio will sustain the link, and the access point’s programming will dynamically adjust the amount of traffic sent to the deteriorated 60GHz radio.   Additionally, the physical design of the AP includes a lip that acts as an umbrella, keeping sheets of rain from coating the 60GHz antenna.

387-picture

The other key and quite snazzy feature of this PTP unit is the self aligning properties of the 60GHz radio, and it’s ability to dynamically scan and realign the link after heavy winds or vibrations.  Inside the access point are two 60GHz antenna elements and a chipset that allows the radio to scan +/- 40 degrees horizontally and +/- 10 degrees vertically in order to acquire or realign the link.

With such a wide scanning angle and the ability to self acquire the link, the need for precision in line of sight deployments is severely lessened.  I love the way Eric Johnson (Director of Product Management for Aruba) put it in his Mobility Field Day 3 presentation, “only minutes to deploy and hold my beer.”

The AP-387 has an extremely narrow (10 degree) beam-width, meaning units could theoretically be placed ~5 meters apart and still use the same channel. PoE+ is recommended for the AP-387, but it will operate with the 60GHz radio backed off by 3 db if 802.3af is used.

It’s important to note that there is a 500 meter distance limitation with these units..The product development reasoning behind the distance limitation is due to size and cost considerations of the high gain antenna elements for the 60GHz radios.  The current unit is rather small in comparison to other 60GHz products in this space, as you can see from the picture below.

AP-387

If you are interested in learning more about this 802.11ad PTP solution, I highly recommend this post  by

Aruba Powering Next Gen Mobility with Eric Johnson and Onno Harms from Stephen Foskett on Vimeo.

 

Disclaimer: While Mobility Field Day, which is sponsored by the companies that present, was very generous to invite me to the fantastic MDF3 event and I am very grateful for it, my opinions are totally my own, as all redheads are far too stubborn to have it any other way.

Published 9/23/2018

 
2 Comments

Posted by on 2018/09/23 in Aruba, Point To Point, Wireless

 

Tags: , , , , , , ,

Cisco Live 2018 – Geek Camp Returns

It’s hard to believe it’s been nearly a month since Cisco Live 2018! The amazing week at Geek Camp was filled with fantastic sessions, Tech Field Day goodness, and lots of great conversations with incredibly talented people.  Engineering Deathmatch had a great lineup, and even launched a new quiz show that was super fun to watch. I am eagerly waiting the posting of those videos. (hint, hint @samplefive).

Seeing long time friends, meeting new folks, and networking with the engineers in the trenches makes Cisco Live an extraordinary experience every year.  I am constantly impressed by a community of people willing to share, mentor, and embrace one another – figuratively and sometimes quite literally, in an effort to educate, help, and support one another in this incredible field. You all know who you are, and you all are phenomenal.

 

 
 

Tags: , , , ,

Upgrading Unity Connection 9.1.2SU2 to 11.5.1SU2

Prologue: This post is intended as an informative prep document for those planning to upgrade Unity Connection (in an HA setup) from 9.1.2SU2 to 11.5.1SU2 using the OS platform upgrade and not PCD (because reasons…) – your upgrade mileage may vary and likely will. These are notes from my upgrade experience and while many of these steps are universal to the upgrade process, each upgrade is a unique snowflake. Do not use this post as a step by step How to Guide. You will be sorry, and I promise TAC won’t be impressed when you tell them that you followed a process from a blog, even one as awesome as mine.

So let’s get started on this upgrade for Unity Connection 9.1.2SU2 to 11.5.1SU2 – the best document to begin with is the Unity Connection upgrade guide found here. Your best bet is to read it through several times and to not be surprised when you can quote any particular paragraph and page number in your sleep.

Also, reviewing the release notes found here is a must.

Once you’ve read the docs and are starting to put together a plan, here’s some guidance on what that plan should entail:

Licensing. You will need some.  Be sure to order your upgrade through PUT and be sure that your licensing server can handle the version of Unity Connection you are going to.  You may need to upgrade your license server or add license definitions to your licensing server pull this off.

Installing a .cop file. Installing a .cop file for RSAv3 keys is listed in the upgrade guide. Digging deeper into this, if you are at version 9.1.2SU2 (AKA 9.1.2.12901-3), this build contains the same fix as the .cop file.

To quote the release notes for this .cop file:

If you are already running CUCM release 10.x or higher you already have this fix and do not need to install this Cisco Options Package (COP) file. If you are running an
Engineering Special (ES)or Service Update (SU)that already contains CSCua88701you do not need to install this COP file. CSCua88701 is included in: 8.5.1.17123-1 and higher, 8.6.2.24122-1 and higher, 9.1.2.11018-1 and higher.

I skipped the installation of this .cop file per this information and also receiving no errors for the .cop results of the “run cuc preupgrade test” output run before the upgrade. For more details, check out this document.

Confirm ESXi support. You will want to be sure the ESXi version your virtual machine is on is supported for the version of Unity you want to be on, and that information lives here.

Determine OVA changes. This step requires checking here and comparing these requirements to your current virtual machine.You will likely need to change your virtual machine vNIC to VMXNET 3 as VMXNET3 is required. You may need to engage a little VMware PowerCLI magic to pull this off.*  Note, if you need more RAM or hard drive space, TAC says a rebuild of the server is required. I’ve heard rumors of engineers changing the memory but the official stance of support is not to touch that stuffs.

Determine your disaster rebuild strategy(s). I recommend downloading and installing COBRAS, getting it running, and exporting all your data from your voicemail servers. With this data, you can import into a fresh build if it comes to that. You should also be sure to have a good DRS backup, which when restored will have all your settings versus COBRAS which will require you to recreate some settings/features.

For me, the basic flow of the upgrade went something like:

Exported data using COBRAS
Confirmed available space on the common partition, need at least 25 GB – I did not have to change high and low water marks, refer to the guide if you need to.
Confirmed nightly backup
Verified voicemail ports registered pre-upgrade
Confirmed HA and replication status pre-upgrade
Ran “run cuc preupgrade test”
Chose not to use utils iothrottle disable command since 24/7 production environment – if I had used this command, after entering it, a reboot is required, re-enabling it post upgrade would also be another reboot
Installed upgrade to publisher, automatic switch version required
Waited for services to establish
Installed upgrade to subscriber, automatic switch version required
Waited for services to reestablish and replication to be good
Powered off pub, ran VMware PowerCLI commands to change vNIC
Powered on pub, waited for services to reestablish
Powered off sub, ran VMware PowerCLI commands to change vNIC
Power on sub, waited for services to reestablish
Confirmed replication and tested voicemail box, call handlers, and MWI functionality
Confirmed all voicemail ports registered to CUCM cluster
Checked RTMT tool for any system error messages
Tested call handlers; tested MWI and messaging waiting indicators
Emailed licensing@cisco.com with generated license request from license server, Active SWSS/UCSS contract number, and PUT order number. Requested only Unity licenses only be migrated.

While this upgrade completed successfully without any downtime, I did run into several issues whose resolutions aren’t exactly satisfying.  If you’ve seen these or have insights into them, please leave a comment, I am sure I’m not the only one who has encountered these and would love to hear if others saw and dealt with them.

After the publisher rebooted when the install completed, the secondary’s web interface was a 404 and would not come back even after a reboot of the secondary. This forced the upgrade to be initiated via CLI for the secondary, but the issue went away in the new version.

Both servers after reboots established voicemail services (voicemail boxes available, call handlers working, etc…) in a matter of 10 – 15 minutes, but watching the RTMT tool, the CM platform services go into a CRITICAL DOWN state for about 30 mins, before eventually settling in an up state. Cisco Unified Serviceability was unreachable during this time, but voicemail services were working. No answer from TAC or forums as to anyone seeing this before and if this is going to cause long term issues.

A red DNS warning shows up in the OS administration and CLI for the servers even though DNS is working fine. Some forums suggest that I may have two entries in DNS forwarding, but that hasn’t panned out to be true. TAC suggests it’s a “cosmetic bug” and that since the functionality is clearly working, not to worry about it – yay?

Had to manually stop and restart replication after the power down/up for vNIC changes, hoping this was an after upgrade fluke, but time will tell.

*If you are looking for some VMware PowerCLI magic for those vNIC changes, I offer this – without support or guarantee, only that these commands did the trick for me.

PowerCLI C:\> Connect-VIServer [esxi IP address]
get-vm “servername”
get-vm “servername” | get-networkadapter
get-vm “servername” | get-networkadapter | set-networkadapter -type “vmxnet3”
Get-View -ViewType VirtualMachine -Filter @{“Name” = “servername”} | %{$_.reload()}

 

Published 1/8/2018

 
13 Comments

Posted by on 2018/01/08 in Unity Connection, Upgrades

 

Tags: , , , , ,

Converting Aruba IAPs to Campus APs for non-UAP access points

Welcome to a quick How To post for converting a shiny new Aruba Instant Access Point (IAP) into a shiny new Campus Access Point (controller based AP). This applies to access points that are not UAP.  For more info on the UAP process and which APs are UAP, check out this awesome post by @theitrebel.

Just power up your access point and once it’s booted, look for an “Instant” SSID.  Connect to this SSID, open a browser, and you will be redirected to a login page* for the controller.  The default login is admin/admin.

Next, click on the Maintenance link, and then navigate to the Convert tab. From the drop down, select Campus APs managed by a Mobility Controller.

Enter the IP or hostname of your controller and click the Convert Now button. Click the confirmation and the conversion will begin. Your AP will reboot, begin anew as a Campus AP, and join the controller you specified. This, of course, assumes you’ve got your controller configured correctly and licensing all in order.

If you’d like to see a short video demonstrating this process, I found this quick video posted by Sean Rynearson: https://youtu.be/Wfl6gYPuyVs

Bonus: If you have a bunch of IAPs to convert, I recommend reviewing this post on the subject, it’ll save you some time: http://community.arubanetworks.com/t5/Controller-less-WLANs/How-to-convert-a-whole-IAP-cluster-as-Campus-APs/ta-p/215053

 

*if you are not automagically redirected when you join the SSID and open a browser window, try manually going to this URL: https://instant.arubanetworks.com:4343/

Published 10/10/2017

 
Leave a comment

Posted by on 2017/10/10 in Aruba

 

Tags: , , ,

Seeing Tetration in action – NFD16

One of the highlights of Network Field Day 16 was a Cisco Tetration presentation by Tim Garner. Launched by Cisco last June, Tetration is a heavy lifting, data crunching platform that soaks up telemetry on all your network packets, uses machine learning algorithms on that data, and produces security policies templates based on the flow information received. This process gives engineers in-depth analytics, an impressive level of visibility, and supplies automagically crafted baseline security policies.  The latter truly shines when you are working with developers and application owners who have absolutely no clue what server needs to talk to what other server(s), much less what ports are required to do so securely.

With Tetration, you can use hardware sensors in the form of Nexus 9K switches with an -X in the SKU, or you can use software agents that can be installed just about anywhere. Or you can use a combination of both.  These sensors look at every single packet going in and out and generate telemetry packets that get shuffled off to Tetration where the real magic happens.

In addition to software agents and hardware sensors that natively generate Tetration metadata packets, you can also stream data from load balancers, firewalls, and other networking devices.  Some devices such as Citrix and F5 are natively supported, but others might take your doing a little work to get the data into a format that Tetration will accept – JSON being one of the acceptable formats.

Another interesting option for getting metadata into Tetration is the use of virtual machines set up as ERSPAN destinations.  Each VM can take in up to 40 gig of traffic, generate telemetry data for this traffic, and stream the data to the Tetration cluster.  Tetration can also take in NetFlow data using this VM method as a NetFlow receiver. NetFlow data is sampled though, so Tetration would not be seeing metadata on every packet as with the other options listed.

Once the data gets to the Tetration cluster, the snazzy machine learning algorithms built into the box start telling you cool things like what hosts are talking to what hosts and what “normal” network behavior looks like, and thereby, what abnormal network behavior would look like.

If your development servers should never be talking to your production servers, Tetration can tell you not only if that what’s happening now, but also if that behavior changes in the future.  Using a Kafka broker* you can have Tetration feed notifications to applications such as Splunk or Phantom, which can in turn communicate with hardware and software devices that perform actions such as host isolation when anomalous traffic is detected.

The automatic whitelists built by Tetration will require some care and feeding by an engineer. Importing policies from ACI is also an option as well. Tetration generated whitelists can be reviewed and tweaked, and an audit of what will be blocked when implementing or making policy changes is an excellent job preserving idea. Checking policies against the four to six months of network traffic data stored by the cluster gives you a good sense of what to expect when enforcement is actually turned on. That being said, you can also run your policies in audit mode for a few months to see what traffic hits the crafted policies.

If you want to see Tetration in action, I highly recommend this video below. The demo starts at about 16 minutes, but Tim Garner is such an excellent presenter, you’ll be glad you watched the whole thing.

 

*Kafka broker service was new to me, basically it’s a notification message bus, I used a few of these links below to get the idea:

https://sookocheff.com/post/kafka/kafka-in-a-nutshell/

https://kafka.apache.org/quickstart

https://www.cloudkarafka.com/blog/2016-11-30-part1-kafka-for-beginners-what-is-apache-kafka.html

 

Disclaimer: While Networking Field Day, which is sponsored by the companies that present, was very generous to invite me to this fantastic event and I am very grateful for it, my opinions are totally my own, as all redheads are far too stubborn to have it any other way.

 

Published 10/6/2017

 
2 Comments

Posted by on 2017/10/06 in Cisco, Network Field Day 16

 

Tags: , , , ,