Introduction
This post will answer the following 3 questions:- How to upgrade Aruba Airwave Server?
- How long does it take to upgrade Aruba Airwave?
- What type and duration of outage can be expected during the upgrade of Aruba Airwave?
The example below shows the actual output from a real upgrade of AMP from version 8.5 to version 8.2.7.1 in a single server environment (no Glass). This specific server is a virtual appliance hosted on Hyper-V.
As Network administrators we are used to upgrades of firmware on a switch, router or WLC taking a few minutes at the most. When it comes time to upgrade the server software we use, we can be caught off-guard by the lengthy amount of time required and it is unclear when any outage will occur. To help with understanding the process I present to you an annotated output of the upgrade process, including timings.
System Output
Key:
System output
My input
My annotations
Outage information
Before starting the backup, download the nightly backup file from the web GUI to a safe location.
T = time of initial login to CLI
login as: ampadmin
Authorised access only. Managed by xyzabc.
ampadmin@server's password:
Last login: Fri Sep 14 13:49:46 2018 from server
Loading Menu...
AirWave Management Platform 8.2.5 on servername
1 Upload File
2 Download File
3 Delete File
4 Backup
5 Restore
6 Support
7 Upgrade
8 Advanced
9 Security
10 Custom Commands
q >> Quit
Your choice: 7
Upgrade
1 Upgrade AirWave Management Platform
2 Upgrade OS Kernel
b >> Back
Your choice: 1
Running Upgrade AirWave Management Platform
AMP version: 8.2.7.1
Running [/usr/local/airwave/bin/start_
Upgrade script AMP-8.2.7.1-amp_upgrade was not found in local cache.
Upgrade package AMP-8.2.7.1-x86_64-cvs.tar.gz was not found in local cache.
Upgrade package will be downloaded from the internet...
Do you use proxy server? (y/N): N
Download upgrade package from:
1. Aruba Support Portal
2. HPE My Networking Portal
Enter your choice (1 or 2): 1
Preparing to connect to Aruba Support Portal...
Enter your Aruba Support Portal username: USERNAMEgoesHERE/PW Prompt not shown
##############################
Upgrade package AMP-8.2.7.1-x86_64-cvs.tar.gz was not found in local cache.
Checking iptables.
T + 2 minutes
Checking the database schema.
T + 5 minutes
Preparing to connect to Aruba Support Portal...
T + 20 minutes (the progress bar stays on a single line unless you hit enter)
T + 20 minutes (the progress bar stays on a single line unless you hit enter)
##########################
T + 32 minutes (this part takes a while)
T + 32 minutes (this part takes a while)
##############################
T + 40 minutes (phew, finally)
T + 40 minutes (phew, finally)
##############################
Validating the upgrade package...
Verifying authenticity of the upgrade package....
Verifying signature....
Good Signature....
Verifying checksum....
Upgrade package verified....
Upgrading AMP to version 8.2.7.1 from version 8.2.5...
Detailed log will be written to /var/log/upgrade/AMP-8.2.7.1-
STEP 1: Moving old version aside.
STEP 2: Unpacking upgrade package.
STEP 3: Checking for compatibility.
T + 45 minutes
T + 45 minutes
STEP 4: Stopping AMP services
T + 46 minutes
46 minutes after starting the upgrade, Airwave Application is unavailable.
The Airwave server itself is still up and responding to ICMP monitoring, and for some time also HTTP monitoring.
As shown in the screenshot below: the browser which was displaying Airwave GUI is still reachable but shows a 502 error in the frame.
T + 46 minutes
46 minutes after starting the upgrade, Airwave Application is unavailable.
The Airwave server itself is still up and responding to ICMP monitoring, and for some time also HTTP monitoring.
As shown in the screenshot below: the browser which was displaying Airwave GUI is still reachable but shows a 502 error in the frame.
STEP 5: Installing upgrade.
T + 55 minutes. Still installing the upgrade. Waiting patiently. No useful feedback on terminal.
******************************
Updated kernel packages that fix various security issues are now
available for your OS. To upgrade, select 'Upgrade' menu item on the AMPCLI Menu,
and then choose 'Upgrade OS Kernel' menu item.
For more information refer to the security advisory:
******************************
T + 57 minutes. Upgrade complete
STEP 6: Restarting AMP services.
******************************
Post upgrade schema check is in progress..
This may take a few minutes..
******************************
T + 59 minutes.
Services are back up (verify by logging in to web GUI). 13 minutes of actual application outage time so far, and server stayed up the entire time. 😏
But wait! There was a prompt to also upgrade the OS Kernel. Looks like we're in for a full reboot after all. 😫.
Services are back up (verify by logging in to web GUI). 13 minutes of actual application outage time so far, and server stayed up the entire time. 😏
But wait! There was a prompt to also upgrade the OS Kernel. Looks like we're in for a full reboot after all. 😫.
Upgrade from 8.2.5 to 8.2.7.1 is successful.
Hit <enter> to continue
Hit enter to continue, 's' to show output, 'r' to show return code.
Upgrade
1 Upgrade AirWave Management Platform
2 Upgrade OS Kernel
b >> Back
Your choice: 2
T + 1 hour 02 minutes
Running Upgrade OS Kernel
A newer version of the kernel is available. If you choose to upgrade
you will need to reboot the system for the change to take effect.
Upgrade the kernel? (y/N) y
Preparing... ##############################
1:kernel-firmware ##############################
2:kernel ##############################
3:kernel-headers ##############################
The kernel has been upgraded successfully.
T + 1 hour 04 minutes. (wow that kernel upgrade was really quick compared to the application upgrade)
Reboot now? (y/N) y
Are you sure? (y/N) y
Broadcast message from ampadmin@servername
(/dev/pts/0) at 10:44 ...
The system is going down for reboot NOW!
Hit enter to continue, 's' to show output, 'r' to show return code.
<end of output>
<end of output>
At this point the server reboots and the SSH session is lost. Now the server is hard down.
T + 1 hour 07 minutes
The server is back up after only a couple of minutes. The services will take some time to start up.
T + 1 hour 10 minutes
Done! The application is back up and running and verified.
Conclusion
The Airwave upgrade process is heavily scripted by Aruba and doesn't need much interaction after starting the upgrade. In fact I only touched the keyboard a handful of times and mostly for y/n input. The upgrade does always take a lot longer than I expect which I why I decided to document it this time.
There were 2 outages: 13 minutes while the application upgraded and another of 6 minutes for the OS reboot.
Despite the fact that I had allocated a 1 hour outage window for this change (which I thought was overly generous at the time, ha), I ran over the window with the second part of the outage actually falling outside of that window, ouch.
Other factors that could possibly impact the upgrade duration are the size of the database and the specs of the VM resources allocated.
Note that the application can be upgraded by firstly downloading the file from Aruba or HPE website then uploading locally to Airwave, then starting the process. I have not yet timed that method.
Note also that the RHEL kernel can be upgraded at any time from the AMPCLI menu, it does not need to happen as part of the application upgrade.
No comments:
Post a Comment
Note: only a member of this blog may post a comment.