How To: Upgrade Aruba Airwave

Introduction

This post will answer the following 3 questions:

1. How to upgrade Aruba Airwave Server?
2. How long does it take to upgrade Aruba Airwave?
3. What type and duration of outage can be expected during the upgrade of Aruba Airwave?

The example below shows the actual output from a real upgrade of AMP from version 8.5 to version 8.2.7.1 in a single server environment (no Glass). This specific server is a virtual appliance hosted on Hyper-V.

As Network administrators we are used to upgrades of firmware on a switch, router or WLC taking a few minutes at the most. When it comes time to upgrade the server software we use, we can be caught off-guard by the lengthy amount of time required and it is unclear when any outage will occur. To help with understanding the process I present to you an annotated output of the upgrade process, including timings.

System Output

Key:

    System output

    My input

    My annotations

    Outage information

Before starting the backup, download the nightly backup file from the web GUI to a safe location.

T = time of initial login to CLI

login as: ampadmin

Authorised access only. Managed by xyzabc.

ampadmin@server's password:

Last login: Fri Sep 14 13:49:46 2018 from server

Loading Menu...

AirWave Management Platform 8.2.5 on servername

  1  Upload File

  2  Download File

  3  Delete File

  4  Backup

  5  Restore

  6  Support

  7  Upgrade

  8  Advanced

  9  Security

10  Custom Commands

  q  >> Quit

Your choice: 7

Upgrade

  1  Upgrade AirWave Management Platform

  2  Upgrade OS Kernel

  b  >> Back

Your choice: 1

Running Upgrade AirWave Management Platform

AMP version: 8.2.7.1

Running [/usr/local/airwave/bin/start_amp_upgrade -f /var/ampcli/user -v 8.2.7.1]...

Upgrade script AMP-8.2.7.1-amp_upgrade was not found in local cache.

Upgrade package AMP-8.2.7.1-x86_64-cvs.tar.gz was not found in local cache.

Upgrade package will be downloaded from the internet...

Do you use proxy server? (y/N): N

Download upgrade package from:

    1. Aruba Support Portal

    2. HPE My Networking Portal

Enter your choice (1 or 2): 1

Preparing to connect to Aruba Support Portal...

Enter your Aruba Support Portal username: USERNAMEgoesHERE/PW Prompt not shown

######################################################################## 100.0%

Upgrade package AMP-8.2.7.1-x86_64-cvs.tar.gz was not found in local cache.

Checking iptables.

T + 2 minutes

Checking the database schema.

T + 5 minutes

Preparing to connect to Aruba Support Portal...

T + 20 minutes (the progress bar stays on a single line unless you hit enter)

##########################                                                36.9%

T + 32 minutes (this part takes a while)

##########################################################                81.1%

T + 40 minutes (phew, finally)

######################################################################## 100.0%

Validating the upgrade package...

Verifying authenticity of the upgrade package....

Verifying signature....

Good Signature....

Verifying checksum....

Upgrade package verified....

Upgrading AMP to version 8.2.7.1 from version 8.2.5...

Detailed log will be written to /var/log/upgrade/AMP-8.2.7.1-upgrade.log

STEP 1: Moving old version aside.

STEP 2: Unpacking upgrade package.

STEP 3: Checking for compatibility.

T + 45 minutes

STEP 4: Stopping AMP services

T + 46 minutes
46 minutes after starting the upgrade, Airwave Application is unavailable. 
The Airwave server itself is still up and responding to ICMP monitoring, and for some time also HTTP monitoring.
As shown in the screenshot below: the browser which was displaying Airwave GUI is still reachable but shows a 502 error in the frame.

STEP 5: Installing upgrade.

T + 55 minutes. Still installing the upgrade. Waiting patiently. No useful feedback on terminal.

***************************************************************

Updated kernel packages that fix various security issues are now

available for your OS. To upgrade, select 'Upgrade' menu item on the AMPCLI Menu,

and then choose 'Upgrade OS Kernel' menu item.

For more information refer to the security advisory:

  https://access.redhat.com/errata/RHSA-2018:2390

***************************************************************

T + 57 minutes. Upgrade complete

STEP 6: Restarting AMP services.

**********************************************

Post upgrade schema check is in progress..

This may take a few minutes..

**********************************************

T + 59 minutes.
Services are back up (verify by logging in to web GUI). 13 minutes of actual application outage time so far, and server stayed up the entire time. 😏

But wait! There was a prompt to also upgrade the OS Kernel. Looks like we're in for a full reboot after all. 😫.

Upgrade from 8.2.5 to 8.2.7.1 is successful.

Hit <enter> to continue

Hit enter to continue, 's' to show output, 'r' to show return code.

Upgrade

  1  Upgrade AirWave Management Platform

  2  Upgrade OS Kernel

  b  >> Back

Your choice: 2

T + 1 hour 02 minutes

Running Upgrade OS Kernel

A newer version of the kernel is available.  If you choose to upgrade

you will need to reboot the system for the change to take effect.

Upgrade the kernel? (y/N) y

Preparing...                ########################################### [100%]

   1:kernel-firmware        ########################################### [ 33%]

   2:kernel                 ########################################### [ 67%]

   3:kernel-headers         ########################################### [100%]

The kernel has been upgraded successfully.

T + 1 hour 04 minutes. (wow that kernel upgrade was really quick compared to the application upgrade)

Reboot now? (y/N) y

Are you sure? (y/N) y

Broadcast message from ampadmin@servername

        (/dev/pts/0) at 10:44 ...

The system is going down for reboot NOW!

Hit enter to continue, 's' to show output, 'r' to show return code.
<end of output>

At this point the server reboots and the SSH session is lost. Now the server is hard down.

T + 1 hour 07 minutes

The server is back up after only a couple of minutes. The services will take some time to start up.

T + 1 hour 10 minutes 

Done! The application is back up and running and verified.

Conclusion

The Airwave upgrade process is heavily scripted by Aruba and doesn't need much interaction after starting the upgrade. In fact I only touched the keyboard a handful of times and mostly for y/n input. The upgrade does always take a lot longer than I expect which I why I decided to document it this time.

There were 2 outages: 13 minutes while the application upgraded and another of 6 minutes for the OS reboot.

Despite the fact that I had allocated a 1 hour outage window for this change (which I thought was overly generous at the time, ha), I ran over the window with the second part of the outage actually falling outside of that window, ouch.

Other factors that could possibly impact the upgrade duration are the size of the database and the specs of the VM resources allocated.

Note that the application can be upgraded by firstly downloading the file from Aruba or HPE website then uploading locally to Airwave, then starting the process. I have not yet timed that method.

Note also that the RHEL kernel can be upgraded at any time from the AMPCLI menu, it does not need to happen as part of the application upgrade.

How To: Using ArubaOS8 API with PowerShell

Introduction

This guide explains:

• the basics of the ArubaOS8 API, 
• specifics of interacting with the ArubaOS8 API when using PowerShell, 
• some general quirks regarding PowerShell usage common to this and other API services 

HPE Aruba's AOS8, starting with v8.0.0, includes a powerful web API which provides both GET and SET capabilities, and also the ability to run show commands which returns structured JSON.

Aruba has published API documentation, which is available from the support site (API PDF).

What struck me with this, as with many API guides, is that the examples are shown only with cURL. I work in and with Windows-centric organisations and I really need to see examples using PowerShell, so I've gone through the guide and selected a few commands to recreate in PowerShell.

Some notes before we begin:

1. On my test device I have a self-signed certificate. If I were using cURL then I could simply use the --insecure flag to skip the certificate check. There is no equivalent switch on PowerShell's Invoke-WebRequest/Invoke-RestMethod commands. I use the method described in the comments of this Stackoverflow page as a workaround. If anyone knows an easier way, please let me know in the comments.
2. All examples below are based on standalone Mobility Controller. A slight variation to the URI is required when connecting to a Mobility Master environment. This is explained clearly in the API docs.
3. A dedicated API local user account with read-only privileges has been configured on the standalone controller.
4. Variables $t1, $t2, $t3 etc. are just variables I use to keep my example outputs separate from one another, as opposed to my other favourite method of calling everything $test ;)

Important disclaimer: I do not claim to be an expert in scripting PowerShell or any other language. At best I'm a tinkerer who can hack enough code together to make my own life easier. Anything copied from here is used at your own risk.

Step 1: Set up some reusable objects

#Set variables

$WLC_IP = '10.13.7.4'

$API_BASE_URI = 'https://'+$WLC_IP+':4343/v1'

$DeviceUsername = "apiuser"

$DevicePassword = "supersecretpassword"



• We've defined the IP address of the AOS8 device. We could also use the FQDN here.
• We've used that IP/FQDN to build a string that will be used in every call.
• We've defined the username and password of the account mentioned in Note 3

Step 2: Run the self-signed certificate workaround

EDIT 13-Oct-18: From Powershell v6 and above (Powershell Core), we now have a new

-SkipCertificateCheck flag available on Invoke-WebRequest and Invoke-RestMethod, so this step can be skipped. This is great news for lab work with self-signed certs. Remember that this is not a substitute for installing a proper cert.

• There are also several other methods to be found with a web search
• As described in Note 1. That StackOverflow link again
• This is only required once per session, but for convenience I save it as a separate .ps script and leave a reference to it in my script, and it runs every time.
• Recommendations for better methods are welcome. Of course the best method is to have a proper TLS certificate installed on the device :)

if (-not("dummy" -as [type])) {
    add-type -TypeDefinition @"
using System;
using System.Net;
using System.Net.Security;
using System.Security.Cryptography.X509Certificates;

public static class Dummy {
    public static bool ReturnTrue(object sender,
        X509Certificate certificate,
        X509Chain chain,
        SslPolicyErrors sslPolicyErrors) { return true; }

    public static RemoteCertificateValidationCallback GetDelegate() {
        return new RemoteCertificateValidationCallback(Dummy.ReturnTrue);
    }
}
"@
}

[System.Net.ServicePointManager]::ServerCertificateValidationCallback = [dummy]::GetDelegate()

Step 3: Log in to the device

#Login to device using previously defined variables
$session = Invoke-RestMethod -Uri "${API_BASE_URI}/api/login" -Method Post -Body "username=$DeviceUsername&password=$DevicePassword" -SessionVariable api_session

• This result of this entire Invoke-RestMethod call will be JSON saved into a variable named $session.
• Take special note of the part at the end where we declared a Session Variable and named it "api_session". We'll be referencing that variable again in every step to follow. This is how PowerShell keeps track of all our web sessions.

Here is what the $session data looks like:

_global_result                                                                                       

--------------                                                                                       

@{status=0; status_str=You've logged in successfully.; UIDARUBA=6cd9028f-f3ea-434d-b3ac-a13385537b0fad}

and here is what the $api_session created on my machine looks like:

Headers               : {}
Cookies               : System.Net.CookieContainer
UseDefaultCredentials : False
Credentials           : 
Certificates          : 
UserAgent             : Mozilla/5.0 (Windows NT; Windows NT 10.0; en-NZ) WindowsPowerShell/5.1.15063.1209
Proxy                 : 
MaximumRedirection    : -1

Step 4: Obtain the UID key and save it for future use

The JSON response of $session data from the previous step can now be interrogated for the security cookie needed for subsequent queries.

#get the UID key session cookie from the login response

$UIDARUBA = $session._global_result.UIDARUBA

We now have an authenticated session, with a cookie. We can now proceed to do stuff!

Step 5: Containers and Objects (complete lists)

Explaining in detail what Containers and Objects do is best left to the formal documentation.
To briefly summarise: We can run a GET against either a Container or an Object. Containers are around a dozen or so groups of objects categories. Objects are specific items, with over 1000 available for query.

• Note how the $API_BASE_URI, the $api_session, and the $UIDARUBA variables all come together now...

#Get complete List of Containers
$t1 = Invoke-RestMethod -Uri "${API_BASE_URI}/configuration/container?UIDARUBA=$UIDARUBA" -Method Get -WebSession $api_session

Get complete List of Objects (may take many seconds to run)
$t2 = Invoke-RestMethod -Uri "${API_BASE_URI}/configuration/object?UIDARUBA=$UIDARUBA" -Method Get -WebSession $api_session

There are way too many Containers and Objects to show here, but I'll show you a snippet of my list:

Containers:

WAN               : @{name=WAN; help=Compression, Health Check, Uplink                                       Management}
LoadBal-Redun     : @{name=Load Balancing & Redundancy; help=Clustering, High Availability, VRRP}
WLAN              : @{name=Wireless LAN; help=AP Group, Client Match, Hotspot, IDS,                                       Mcell, Mesh, Mobility, RF, SSID,                           
                                Virtual AP}
Services          : @{name=Services; help=ALE, Airgroup, Lync, Openflow, SDN}
AP-Provisioning   : @{name=AP Provisioning; help=AP Provisioning, AP Whitelist,                                       Provisioning Profile}
Unknown           : @{name=Unknown; help=Fix these objects, catchall container}
Interfaces        : @{name=Interfaces; help=Physical/Logical/Loopback Interfaces,                                       Tunnels and USB/Modem Interfaces}

Objects:

      "arm_error_rate_threshold": {
        "error-rate-threshold": {
          "_min": 0, 
          "_type": "INT", 
          "_max": 100, 
          "_default_val": 70, 
          "_help": "% min rate for error in channel that triggers a channel change. Default 70. Recommended value 70"
        }
      }, 
      "arm_error_rate_wait_time": {
        "error-rate-wait-time": {
          "_min": 0, 
          "_type": "INT", 
          "_max": 2147483647, 
          "_default_val": 90, 
          "_help": "Minimum time in seconds error rate has to be high to trigger a channel change. Default: 90."
        }
      }, 
      "channel_quality_aware_arm": {}, 
      "arm_channel_quality_threshold": {
        "channel-quality-threshold": {
          "_min": 0, 
          "_type": "INT", 
          "_max": 100, 
          "_default_val": 70, 
          "_help": "Channel quality below which triggers a channel change. Default 70%."
        }

We now have the complete lists of Containers and Objects. Educational but not very useful in itself, so on to the next step.

Step 6: Some specific examples of Containers

We've seen the list of all containers, now lets get a couple of specific containers and extract some specific information from them

#Get Info of specific Container (example: Interfaces)
$t3 = Invoke-RestMethod -Uri "${API_BASE_URI}/configuration/container/Interfaces?UIDARUBA=$UIDARUBA" -Method Get -WebSession $api_session

#Get Info of specific Container (example: Crypto)
$t4 = Invoke-RestMethod -Uri "${API_BASE_URI}/configuration/container/Crypto?UIDARUBA=$UIDARUBA" -Method Get -WebSession $api_session

• We've grabbed all the info from the Interfaces container
• We've grabbed all the info from the Crypto container
• Woohoo, now we are getting some useful data about our device!

Let's check that Interfaces JSON for VLAN info:

#Example: show VLAN Names
$t3._data.vlan_name_id



Here is the VLAN info on my system:

name              vlan-ids

----              --------

Corp-SSID           241     

BYOD-SSID        244     

Guest-Legacy       100     

Corp-Legacy        28      

Guest-SSID          242     

FrontageMain       20 

Step 7: Some specific examples of Objects (with optional filtering)

Note: The syntax for filtering can get a little complicated, please refer to the full API documentation for details on that.

#Get Info of specific Objects (example: Int VLAN)
$t5 = Invoke-RestMethod -Uri "${API_BASE_URI}/configuration/object/int_vlan?UIDARUBA=$UIDARUBA" -Method Get -WebSession $api_session


#Get Info of specific Objects with filter (example: Int VLAN, filter IP addr and MTU)
#URI Needs a bit of massaging due to so many nested quotation marks. There is definitely a better way to do this!
$t6uri1 = '/configuration/object/int_vlan?&filter=[ {"OBJECT" : { "$eq" : ["int_vlan.int_vlan_ip", "int_vlan.int_vlan_mtu"] } }  ]&'
$t6uri2 = "${API_BASE_URI}${t6uri1}"
$t6 = Invoke-RestMethod -Uri "${t6uri2}UIDARUBA=$UIDARUBA" -Method Get -WebSession $api_session


We can compare the responses and clearly see that the filtering is effective:
PS C:\> $t5._data

int_vlan                                                                                                                                                                             
--------                                                                                                                                                                             
{@{id=1; int_vlan_shut=; int_vlan_routing=; int_vlan_ndra_hlimit=; int_vlan_ndra_interval=; int_vlan_ndra_ltime=; int_vlan_ndra_mtu=; int_vlan_nd_reachtime=; int_vlan_nd_rtrans_t...



PS C:\> $t6._data

int_vlan                                                                                                                            
--------                                                                                                                            
{@{id=1; int_vlan_mtu=}, @{id=20; int_vlan_ip=; int_vlan_mtu=}, @{id=240; int_vlan_ip=; int_vlan_mtu=}, @{id=241; int_vlan_mtu=}...}

There is a bunch of info available about the interfaces. Next we use regular PowerShell syntax to view the one or all interfaces:

PS C:\> $t5._data.int_vlan[1]


id                              : 20
int_vlan_ip                     : @{ipaddr=10.13.7.4; ipparams=ipaddrmask; ipmask=255.255.255.0}
int_vlan_routing                : @{_present=True; _flags=}
int_vlan_ndra_hlimit            : @{_flags=; value=64}
int_vlan_ndra_interval          : @{_flags=; value=600}
int_vlan_ndra_ltime             : @{_flags=; value=1800}
int_vlan_ndra_mtu               : @{_flags=; value=1500}
int_vlan_nd_reachtime           : @{_flags=; value=0}
int_vlan_nd_rtrans_time         : @{_flags=; value=0}
int_vlan_mtu                    : @{_flags=; value=1500}
int_vlan_suppress_arp           : @{_present=True; _flags=}
int_vlan_ip_ospf_cost           : @{_flags=; value=1}
int_vlan_ip_ospf_dead_interval  : @{_flags=; value=40}
int_vlan_ip_ospf_hello_interval : @{_flags=; value=10}
int_vlan_ip_ospf_prior          : @{_flags=; value=1}
int_vlan_ip_ospf_retransmit_int : @{_flags=; value=5}
int_vlan_ip_ospf_transmit_delay : @{_flags=; value=1}



Let's finally see some IP addresses of this box:

#Example: show int VLAN IP from unfiltered response
$t5._data.int_vlan[1].int_vlan_ip

PS C:\> $t5._data.int_vlan[1].int_vlan_ip

ipaddr        ipparams   ipmask       
------        --------   ------       
10.13.7.4   ipaddrmask 255.255.255.0

#Example: show int VLAN IP from filtered response
$t6._data.int_vlan[1].int_vlan_ip

PS C:\> $t6._data.int_vlan[1].int_vlan_ip

ipaddr        ipparams   ipmask       
------        --------   ------       
10.13.7.4   ipaddrmask 255.255.255.0

Step 8: Running any 'Show' command

A neat feature of this API is the ability to run any show command, and get the result in a somewhat more structured format than a simple screen-scrape. 

Here is how that looks:

#Running any show command (example: 'show iap table')
$t7 = Invoke-RestMethod -Uri "${API_BASE_URI}/configuration/showcommand?command=show+iap+table&UIDARUBA=$UIDARUBA" -Method Get -WebSession $api_session
#Example: show Up/Down branches
$t7._data
#Example: show the table data
$t7.'IAP Branch Table'

We've now got two forms of data about the 'show iap table' command we sent.
Viewing $t7._data gives us:

PS C:\> $t7._data
Trusted Branch Validation: Disabled
Total No of UP Branches   : 2
Total No of DOWN Branches : 0
Total No of Branches      : 2

While viewing $t7.'IAP Branch Table' gives us:

PS C:\> $t7.'IAP Branch Table'


Assigned Subnet : 
Assigned Vlan   : 242,244
Inner IP        : 10.25.2.11
Name            : SiteA-VC
Status          : UP
VC MAC Address  : 20:a6:cd:aa:bb:cc

Assigned Subnet : 
Assigned Vlan   : 242,244
Inner IP        : 10.25.2.10
Name            : SiteB-VC
Status          : UP
VC MAC Address  : 20:a6:cd:dd:ee:ff


What if we just want the status of the SiteB tunnel? Then we simply use $t7.'IAP Branch Table'[1].Status which gives us:

PS C:\> $t7.'IAP Branch Table'[1].Status
UP

Step 9: Logging out

Practice good hygiene. Always remember to log out!

#logout
Invoke-RestMethod -Uri "${API_BASE_URI}/api/logout"  -WebSession $api_session

The device replies with a friendly message confirming the log out:

_global_result                                                               

--------------                                                               

@{status=0; status_str=You've been logged out successfully.; UIDARUBA=(null)}

• Note that the structure of the logout command is different to that of the GET requests, and the UIDARUBA security cookie is not required.

Conclusion

These have been a few examples of API GET requests using the AurbaOS8 API and PowerShell. 

There is plenty more that can be done with the same API e.g. SET operations, counts, special pagination, viewing pending vs actual data, adding users, a special 'write mem' object, and more. Once again, refer to the officially published API docs and explore the Objects catalogue to understand the full feature set available.

I hope this information has been useful to my fellow PowerShell users. I have already put it into practice for monitoring a specific item that has no SNMP OID attached.

Please leave your comments below.