RSS

Minor version upgrades for Aruba controllers, version 6.5.x

If you tuned in for the last post, you’ll remember that in all the wireless mesh troubleshooting fun, a wireless controller upgrade was required.  Today’s post outlines the upgrade process from 6.5.0.1 to 6.5.1.7 with a primary and secondary controller. As always when dealing with upgrades, your mileage may vary. Never forget to read the release notes, and when in doubt, contact TAC.

As with any upgrade, the release notes often contain subtle details, and these are typically details that bite back if you miss them.  The notes for 6.5.1.7 are pretty straightforward, but they do include exciting, not to be missed, caveats if upgrading from 6.4.x, as well as some solid tips for the upgrade.

The best practice advice in the notes includes steps such as confirming enough memory and storage space for the controller, making a backup of the configuration, and noting the number of active access points before upgrading. All of these suggestions make prudent sense and the commands to do so are listed in the guide.

You can use a FTP, TFTP, SCP, local file, or USB to do this upgrade, but the guide warns against using a Windows-based TFTP server. I used FileZilla FTP server.

Once you’ve downloaded the image file from Aruba and your pre-upgrade checklist is complete, navigate to Maintenance > Controller > Image Management -> Master Configuration.

Pick the file loading option you want to use for the upgrade, then fill in the required details for the transfer. Choose the non-boot partition for Partition to Upgrade. This will load the new software image to the inactive partition. If you are uncertain which one is the boot partition, look under Current Partition Usage, one partition will be marked **default boot**. You will want the other one.

Be sure that Reboot Controller after Upgrade is set to No, unless you have a single controller and eat danger for breakfast. Select Yes for Save Current Configuration Before Reboot, and click Upgrade.

At this point, you rinse and repeat for the other controller(s).  Once the controllers have the upgrade version loaded, you reboot the master, and simultaneously reboot the other controller. In voice upgrade world, you have been well trained to wait FOREVER for all the services to come back up on the primary before even considering a reboot of secondaries, but in Aruba wireless world, simultaneous is written in the guide. See excerpt below from the 6.5.1.7 Release Notes available on the Aruba support site.

TAC did ease my anxiety over this simultaneous reboot thing by letting me know no harm would be caused if I wanted to wait for the master to come back online completely before proceeding.

After the controllers reboot and are back online, access points begin getting their new firmware and rebooting. Once the dust settles, you should end up with the same number of active APs as you noted in the beginning. Then it’s all testing and confirming access points are happy, clients are connecting, and that all is well your WiFi world.

Published 9/11/2017

Advertisements
 
2 Comments

Posted by on 2017/09/11 in Controller Upgrades, Wireless

 

Tags: , ,

Wireless troubleshooting – mesh access points stop accepting clients

Today’s topic initially spawned from mesh troubleshooting.  If you’ve worked with mesh much, you may have just thrown up a little and are frantically trying to close the browser tab that led you here, and that’s totally understandable.  For my voice engineer readers, mesh troubleshooting is one of the few things in the universe that can generate pain and suffering in levels akin to troubleshooting fax machines.

Given typically vague rumors and incomplete reports of intermittent connectivity issues at a mesh site, my amazing coworker was able to hone in on the root problem: various APs in the mesh intermittently stopped accepting clients on their 2.4 GHz radios. Being as 5 GHz was limited to back-haul traffic only, this was definitely wreaking some havoc on the overall deployment.

From the controller, disabling and re-enabling the radio on the 802.11g radio profile* for the affected APs served as a workaround while TAC was consulted. Mysteriously, other mesh deployment sites with the same model AP and code were not seeing this issue. As a note, these APs were all Aruba 274s and controller code version 6.5.0.1, but spoiler alert, the model AP wasn’t the issue.

Fast forward to TAC and some show/debugs commands later, the APs that randomly stopped accepting clients had enormous amounts of 2.4 radio resets, indicating this bug discussed here.

This issue affects not only 274s, but other models of access points. The bug does not appear to affect all of any model, just a small percentage of access points when it does show up.

If you think you might be experiencing this issue, take a look at the output of these commands and look for a crazy high number of radio resets.  How high?  Since the radio appears to be resetting practically every second, the radio reset number is noticeably and ridiculously large.**

show ap debug radio-stats ap-name NAME-OF-AP radio 0 advanced
show ap debug radio-stats ap-name NAME-OF-AP radio 1 advanced
show ap debug system-status ap-name NAME-OF-AP

The fix is in code versions 6.5.0.4 or 6.5.1.3 or later.  We landed 6.5.1.7 and the issue looks to be properly squashed. The upgrade process, which I’ll outline in another brief post, was a simple and straightforward, and being a veteran of many a lengthy and challenging voice upgrades, I found this to be refreshingly delightful and far less sanity taxing.

* Enabling the radio can be done on the 802.11g radio profile, on the Basic tab.  Uncheck the Radio enable box, click Apply.  Check the Radio Enable box, click Apply.  These mesh APs each have their own AP specific profile and this action only affects the individual AP.  If your AP doesn’t have an AP specific profile, be sure to know what APs you are impacting when you do this. Also of note to this case, some experiencing this issue found disabling ARM provided temporary relief, but didn’t do the trick in this deployment, as ARM was already disabled and the issue still occurring.   

Radio enable

**Below is an example of the number of resets seen for one of the affected APs:

Interface Rx_pkts Rx_errors Rx drops Tx_pkts Tx_errors Tx_drops Resets
——— ——- ——— ——– ——- ——— ——– ——
wifi0 174210795 15727807 0 451900531 103 0 9
wifi1 9166677 133711103 0 32655175 842870 0 211094

 

Published 09/05/2017

 

Tags: , , , , , , ,

Cisco Live 2017, engineering awesomeness.

Spending a week with amazing engineers always ranks high on my list of reasons to attend Cisco Live every year.  The networking community and the behind the scenes work of the Cisco Live team make this event truly fantastic every year, and 2017 was a definitely a hit.

I especially enjoyed participating in Tech Field Day once again.  OpenGear presented Lighthouse 5 which focuses on automating setup and maintenance, leveraging new API goodness. OpenGear’s API aims to enhance scale of deployments, while streaming workflows.  I found it especially fun watching Slack be leveraged to enroll and communicate with the OpenGear device. Nerdy goodness I recommend checking out.

If you are looking for a monitoring solution, I highly recommend you check out this excellent PRTG demo by Benjamin Day of Paessler, who not only knows his stuff, but refuses to use even one Power Point slide for his Tech Field Day presentation.  The man is a genius. The PRTG notification enhancements, maps, and overall flexibility really stood out, definitely cool stuff. You won’t be sad you watched.

And in the final bit of Tech Field Day learning for me, NetApp’s presentation on their FlexPod SF solution took a room full of network engineers and captivated their attention on storage. I know it sounds hard to believe that network engineers could find a storage presentation fascinating, but Jeremiah Dooley managed to pull off this incredible feat, and I highly recommend checking out this session.  He covers all the important details of the FlexPod SF announcement, including the available architectures, in a way that makes network engineers forget that this is a solution focused on storing bits, and not just moving them.

The return of Engineering Deathmatch to Cisco Live featured several episodes with some of my fabulous (and lovingly voluntold for EDM) friends, who couldn’t be more amazing. I’m excited to check out the Engineering Deathmatch site as the episodes air over the coming weeks.

And lastly, my favorite annual tradition of Cisco Live wrap up blogging, the photo gallery of crazy, brilliant, hilarious engineers being remarkably phenomenal. I heart you all.

Published 07/04/2017

 
2 Comments

Posted by on 2017/07/04 in Cisco Live 2017

 

Tags: , , , , , ,

HPE Discover 2017, Las Vegas

Attending HPE Discover 2017 did not disappoint. It was a fabulous week filled with presentations from subject matter experts on cool new tech, conversations with incredibly talented engineers and bloggers, and maximum levels of geeking out with other geeks.

I suspect this blog audience would be super interested to hear more about the new 8400 Aruba core switch announced at HPE Discover this year.

The speeds and feeds, along with and all the usual data sheet info is here, but what really stands out is the emphasis on telemetry data and programmability. Much of the focus on visibility and automation has been leveraged to make troubleshooting easier for the engineer.

The demonstration I saw up close was a simple script that allowed for monitoring of the priority voice queue. The script automagically detected any issues with the queue, captured offending packets when there was an issue, and presented the info to the user.  The Network Analytics Engine even gave some guesses as to why the issue occurred.  The demo I saw is pretty similar to what you can see in this short demo.

The 8400 is the first core switch Aruba has come out with, and it touts a new OS based on the existing Aruba switch OS. Yes, the thought of a new OS makes me a tad nervous when talking core switching, so be sure to check out the Coffee Talk Day 2’s first session in which the thoroughness of the OS testing process is discussed. If you’d rather not watch the whole thing, just know that code quality is a focus of the developers involved.

Other cool HPE Discover announcements included Aruba Asset Tracking, which leverages BLE enabled tags and Meridian Location Services to keep up with your stuff in real time. Data sheet goodness is here – see excerpt below from the data sheet to see the APs that support Asset Tracking.

For more HPE Discover 2017 goodness, check out these recorded sessions, I especially recommend Day Three’s talk on machine learning algorithms and the state of AI, completely fascinating, totally nerdy goodness.

Coffee Talks Day 1
Coffee Talks Day 2
Coffee Talks Day 3

Disclaimer: While HPE was very generous to invite me to this great event, my opinions are totally my own, as all redheads are far too stubborn to have it any other way.  Also, special thanks to Pegah, Laura, and Becca for doing such a great job organizing this event.

 
1 Comment

Posted by on 2017/06/20 in HP Networking, Uncategorized

 

Tags: , , , ,

Oracle Ravello Blogger Day, 2017

Attending Oracle Ravello Blogger Day last month provided me deep insight into two products I knew little about before attending, Oracle Cloud and Ravello.  After the excellent deep dive provided and the basic melting of my brain on all things hypervisor, virtualization, and cloud, crafting an intelligible post seems a formidable challenge. But here we go:

Oracle has a cloud?! Yup. And they are pretty serious about where they are taking this. Over the last three years, there’s been a serious commitment to time and resources to build this thing and to build it right. Clay Magouyrk, VP of Oracle Cloud Infrastructure, jokingly commented one best things about being late to the cloud game is learning from other peoples mistakes.  Cloud isn’t new and watching what is working for the market leaders and avoiding their pitfalls is practically industry tradition.  But there’s differentiation here as well, with Oracle touting non over-subscription, predictable latency, bare metal access, and competitive pricing.  The Oracle cloud still has construction work to be done – only two US regions (think availability zones) are available at this time, but a European region is soon to be established.

Ravello, what is is? Ravello uses nested virtualization to allow you to bring your VMware based applications into the cloud without changing anything about them.  It reads the metadata of your virtual machines, sets up your virtual networking for you, and presto! You have your VMware environment running on cloud infrastructure.  Why is this handy?  Well, lots of vExperts have already leveraged this for their studies and lab environments.  Being able to test large scale scenarios without laying out great big wads of cash into your own virtual infrastructure is huge. For you networkers, this reminds me of Forward Networks where you basically have an accurate running copy of your network that you can break as you will. My favorite case study presented at Oracle Ravello Blogger Day was a network security company whose Ravello template, comprised of hundreds of endpoints and servers, is used to train engineers using true-to-life malware incidents.

Why Ravello and Oracle Cloud together? Ravello has in the past been cloud agnostic and still plans to stay that way, but there will be added benefits if you chose to run Ravello on Oracle cloud – those benefits stemming from the ability of Ravello developers to tap into the underlying infrastructure and eek out that extra bit of performance.  I would try to explain the hypervisor intricacies that allow this dark magic to happen, but I would quickly resort to words like abracadabra and shibboleet.

Fortunately, many of my vExpert friends have already blogged on the finer details of Oracle Cloud and the Ravello announcements and I highly encourage you to check these out:

Chris Wahl (@chriswahl): Getting Nerdy on the Oracle Ravello Cloud Service

Ather Beg (@atherbeg): Oracle Ravello Blogger Day – Part 1: Oracle Cloud Oracle Ravello Blogger Day – Part 2: Ravello

Gareth Edwards (@garethedwards86): Ravello 2017 Bloggers Conference – Opening Post #RBD1

Max Mortillaro (@darkkavenger): RT1 – Oracle Cloud Strategy: Part 1 – Oracle Ravello Cloud Service

Matt Leib (@MBLeib): Ravello Systems Changing the Game

James Green (@jdgreen): Can Oracle Build a Viable Public Cloud

Keith Townsend (@CTOAdvisor): Oracle’s Cloud Investment is Real

Tim Smith (@tsmith_co): Ravello and the Oracle Cloud Journey

 

Disclaimer: While Mark Troyer and the awesome folks at Teck Reckoning were very generous to invite me to this fantastic event which was awesome, my opinions are totally my own, as all redheads are far too stubborn to have it any other way.

 

Published 06/02/2017

 

 

 
1 Comment

Posted by on 2017/06/02 in Oracle Ravello Blogger Day

 

Tags: , ,

Enforcing wireless SSID policy using CounterACT NAC and Airwatch MDM module

Recently, I tested out policy enforcement on corporate iPads using Forescout’s CounterACT and its optional Airwatch integration module*.  I’ll be sharing a few things I learned along the way, especially since documentation of this setup is rather sparse (read practically non-existent).

To get this setup, I installed the Airwatch module, downloaded from the Forescout site. You’ll need a valid login, but once you’ve downloaded the file, you can install the module from the CounterACT client by going to Tools -> Options -> Plugins.  After installing the module, there’s a few pieces of integration information that can be found in the Airwatch portal itself.  In the CounterACT client, you can right-click the AirWatch MDM plug-in, click Configure, and enter the required information.

Once you’ve completed the integration information, be sure to start the AirWatch MDM Plugin – it doesn’t automagically start and results are particularity disappointing unless it’s running, as I experienced myself.

At this point it is a good idea to use the Test option for the Plugin and confirm you see a Test Passed in your output.

You can also double click the MDM Integration Module and you should see some happy little Airwatch managed devices listed.

Now it’s time to set up a couple of policies. My first policy matched on Network Function – Mobile Device and Airwatch Enrollment Status – Enrolled.  If CounterACT finds these two criteria to be true, it should drop the tablet into my Corporate Hosts group I designated – a group which is allowed the appropriate network access for a corporate managed device.

My second policy was designed to match unmanaged tablets and phones – those not enrolled in Airwatch. The policy checks if the Device Function is Mobile Device, and has an action of WLAN Block.

I thought this would be it and victory would be mine, but alas the WLAN block wasn’t working.

I received increasingly annoying errors about not being able to reach the wireless controller to enforce the policy.  After testing the wireless controller under Plugins, I could see I was failing on WLAN Role and on Write Permissions.   In an act of sheer grasping at straws, I removed the wireless controller, which had been added as it’s VIP (HA) address, and instead added the wireless controller as its “real” IP address. That did the trick. All tests passed, victory dance cued up.

But then I discovered the extremely disturbing flaw in my perfect policy plan.  Once a device was identified as not managed and very successfully blocked, it became blocked from all SSIDs. Meaning no employee network AND no guest network for the device. The controller wouldn’t accept the device as a client. Period. Cue sad trombone instead.

Reworking the policy logic, I instead implemented the WLAN Role change instead of the WLAN Block action.

I selected a guest role configured in the Aruba controller that locks the user down, blocking access to corporate resources.  You can see below the successful policy enforcement.

Later, my awesome coworker was able to set up an HTTP notification action for the policy so that the users see a web page informing them of the error of their ways and instructing them to change SSIDs to the guest wireless network to be redeemed.

 

Now about that victory dance…

 

Published 05/26/2017

*Anyone who has had to deal with 802.1X certificate enrollment for iPads knows what a PITA the experience is – the setup being tested here allowed for PEAPv0 (EAP-MSCHAPv2) authentication using Microsoft NPM, with the goal that any non-corporate device would have no access to corporate resources. There are other ways to skin this ugly iPad certificate cat, and if you’d like to list the ones you’ve had the most success with in the comments, I am sure others would appreciate the insight.

 

 

 
Leave a comment

Posted by on 2017/05/26 in NAC, Tools, Wireless

 

Tags: , , , , , , ,

Capturing 802.11 management frames on Windows using Acrylic WiFi Pro

Studying for CWAP, I embarked on a mission to capture 802.11 management frames using my Windows laptop. For those with MacBooks that do this natively, read no further, just keep on perfecting that smug look of disdain with a slight hint of pity for the rest of us Microsoft peasants.

For those whose laptops aren’t fruit branded, but you still want to capture 802.11 frames in promiscuous mode, this is the post for you.  Especially if you can’t quite justify the cost an AirPcap adapter for study purposes.

While researching alternatives to pricey AirPcap adapters, I came across this Acrylic WiFi Professional post on their option for an NDIS driver. This driver allows you to capture in promiscuous mode, so you can capture all that management frame goodness, but without the AirPcap adapter.  I checked out the supported USB wireless options, ordered one off the list from Amazon (I picked the NETGEAR A6200), and downloaded a free trial of Acrylic WiFi Pro to get started.

The installation of Acrylic Pro is straightforward, as is turning on Monitor Mode when you know where to look. By default, Monitor Mode is turned off and the NDIS driver is not installed.  Just click the menu in the right corner, and select Change to get to the Monitor Mode settings.

mode

 

Select Monitor Mode On and select Install the NDIS driver.  You’ll get a warning message that you might crash your system and you’ll need to acknowledge that you are completely okay with this*.

NDIS warning message

 

Once the driver is installed you can swap over to the Packet Viewer using the icon in the top tool bar or by clicking Packet Viewer from the menu.  You will also see that you are in Monitor Mode and can select to change out of Monitor Mode if so desired.

Packet Viewer Window

 

While all of this is really super cool, I was extremely  interested in capturing these frames inside of my most familiar tool of packet sniffing choice, Wireshark.

Unfortunately, I didn’t see the NDIS driver as an available capture interface when I launched the Wireshark application. This post by Acrylic reminded me why. I needed to launch Wireshark with Run As Administrator, even though I am a local administrator on the laptop**.   Once I did this, I could select the Acrylic NDIS NETGEAR A6200 WiFi Adapter and start capturing wireless management frames.

Wireshark Capture Interfaces

 

I could also select the Wireless Toolbar in Wireshark and see that the NDIS driver emulating an AirPcap adapter.

wiresharkwirelessmenu

wiresharkwirelesstoolbar

 

Unfortunately, I still had one tiny problem at this point.  Every time I launched the Wireshark application, my built-in wireless card immediately quit passing all traffic. Not exactly ideal for productivity.

Easy fix, though, if you encounter this issue.  Head over to the settings for the Network Adapter, uncheck the Tarlogic NDIS Monitor Driver for the built-in adapter, and the problem is solved.

Change Adapter Settings

I would be remiss not to point out that there are limitations to this NDIS driver. For instance, there is no support for 40 or 80 MHz channels at this time.  But for my CWAP study purposes, this is working quite well and saves me a bit of cash.  Also, Ben Miller did a great write up on this very same subject, which, of course, I found just AFTER I went through this process and drafted this post. The universe has quite the sense of humor like that.

 

Published 2/7/2017

*Do this at your own risk, please don’t blame me for your system crash, there’s a good chance I’ll just point and laugh…

**If you need to know how to set a program to always run as administrator in Windows 10, look here.

 

 

Tags: , , , , , , , , , , ,