I recently had a customer sign a HPE GreenLake dHCI contract that included a new Nimble AF40 to replace a 4 year old AF20Q, and we completed the migration this past week. As a MSP, a critical part of the deployment services that we provide our clients includes the sanitizing (or secure wiping) of any products we replace, and this AF20Q was no exception, other than it’s the first 5th generation Nimble we’ve decommissioned. HPE does provide a KB article in Infosight on the necessary steps (search for “KB-000361 Array Sanitization”). And while it is a good guide, it doesn’t totally cover everything you should know to be prepared for prior to starting this process. I should also note, that if you run this sanitization process, when you are done the Nimble becomes nothing more than a boat anchor as it will not have any sort of OS on it to boot from. If you plan to repurpose or reuse it after sanitizing, you will need to reinstall the Nimble OS from two USB keys that you can obtain from Nimble Support. I’d like to share my experiences in this process in the hopes that it will save someone else a headache (both literally and figuratively).
If your Nimble is running an OS version earlier than 5.0.3.0, then these instructions are not for you! You will need to contact Nimble support to obtain two bootable USB keys with the sanitization process on them. If your Nimble is running OS 5.0.3.0 or newer, then you should be good to go with these steps as I’ve written them out.
And as always before I begin:
Use any tips, tricks, or scripts I post at your own risk.
We’ll start with the obvious. Make sure the array your are sanitizing is the correct array, that it’s had all the hosts disconnected from it, and that it has all the volumes / data removed from it in the Nimble’s management console. And of course, ensure the customer actually want’s it sanitized!
In my case, after completing the migration from the AF20Q to the customer’s new AF40, I removed the array from the customer’s rack, brought it back to my office and set it on my workbench, which is 6 feet away from my desk. This was my first mistake – once you boot the unit to begin sanitization, the fans kick up to high, and they don’t stop! This AF20Q had 12 x 960GB SSDs in it, and the sanitization process took about 8 ¼ hours from power on to power off. With the fans running at high! Needless to say, we found a way to move this unit into another room without powering it down (not recommended, but I had already started the sanitization process and couldn’t stop) and without unplugging a single cable or moving my notebook off the top of it! So if you have a Nimble to sanitize, think very carefully about where it is going to be sitting and the prolonged noise levels that are going to occur during the sanitization process!
Next, when I talked to Nimble support prior to starting the sanitizing, they warned me not to close the serial console sessions to the array once I start the process until it ends, otherwise you’ll have no way to monitor the process (and you’ll never actually know if it completed or not). I would strongly recommend you utilize a dedicated notebook for the task and disable all power management, screensavers, lock screens, and obviously plug it into power with its power brick.
The picture below is after I had finished because as you can see the array has no power cables plugged in and my notebook is powered off.
You should find attached to the back of the Nimble a pair of male DB9 to 3.5mm plug serial adapters (the 3.5mm plug looks like wired headphone jack). You’ll need these, along with two USB to Serial (DB9 – male) adapters, and two DB9 (female / female) serial cables. Plug both USB to Serial adapters into the notebook, and open Windows’s Device Manager to determine the com port numbers associated with each. Plug the other end of the USB to Serial adapter into the DB9 f/f cable and the other end of the DB9 f/f cable into the DB9 to 3.5mm serial adapter. The 3.5mm plug goes into the back of each controller.
With the Nimble still booted up and running, open Putty, and create a serial connection (115200, 8, N, 1) using the correct com port to Controller A. Then open Putty again and create a second connection using the correct com port to Controller B. When you hit enter in each Putty session, you should get a login prompt from the array. Don’t bother trying to login though (it wouldn’t accept the array’s admin credentials when I tried it). All you care about at this time is that you have connectivity via the serial console and can see the console output and that it responded to the keyboard input. The reason you are doing this with the array still booted and running is to verify you can actually see the serial console output via your serial adapter setup – my initial attempt didn’t give me console output and I had to go find a different serial cable. Had I not tested this in advanced, I probably would have gotten impatient and started messing with the settings and stuff before the serial console started outputting any text during the boot process and I would have missed it.
Now you need the Sanitization key (password). For this – you need to call Nimble Support (1-877-364-6253 if you are North American based) and they will generate a Sanitization key which they will email to you. Support told me the key is valid for 24 hours, although KB-000361 says it’s closer to 48 hours – either way, once you have the key, you are on a clock to start.
**Note – I do not know if the array must have a valid support contract on it for them to give you a key – this one still had a contract on it.**
Once you have the key and are ready, open Putty and make your two separate serial connections. I strongly recommend you arrange the two sessions on your screen so they are literally side by side, top to bottom (see picture further below), because you are going to need to be able to see the output of each simultaneously, and you are going to need to interact with both side by side simultaneously (trust me – you do not want to be alt-tabbing trying to find these sessions in the 3 seconds you have to make the correct selection in each – close everything else up except these two sessions!). Open a 3rd Putty session (it can be over top your two serial connections for now though) and this time SSH the management IP of the array and login as admin.
You are now going to reboot the entire array (so both controllers), **AND** (this is important), you must be ready to jump from one serial console to the other at almost the same time to select the correct boot option once it reboots (so get your coffee and use the washroom if you need before you reboot the array instead of thinking you can do so while it’s restarting)!
In order to reboot the array, you need to know the array’s name. To find this, in the SSH session (logged in as the Nimble admin user account) type:
array --list
Then, to reboot the array, you are going to type:
reboot --array array_name
Answer the confirmation for reboot, and after a few seconds, your SSH session will terminate. Close this terminated SSH Putty window and switch to your two Putty serial console sessions (again, they should now be side by side on your screen as shown below). You probably won’t see any activity at all for what seems like 2 or 3 minutes in these sessions. Then suddenly in both sessions (one may be slightly slower than the other – in my case Controller A was about ½ second behind Controller B in all these steps) you’ll see the “Intel® iSCSI Remote Boot” prompt appear for 2 or 3 seconds, and the screen will clear and a few seconds later you’ll see an Intel BIOS screen with [ENTER], [F2], [F6], [F12] options at the bottom of the text.
Get prepared to act fast! A few seconds later this too will disappear and you’ll see a normal grub boot loader page appear with 3 boot options. In the Putty session this appears in first, hit the down arrow twice to move to the 3rd option – “Nimble Array Appliance Sanitization Boot” and hit enter (to select it), then very quickly move to the other Putty serial session and do the same thing. Your total window of opportunity for this is about 3 seconds for both Putty sessions combined – if you miss it (on either controller), immediately pull the power cables from the array, wait 15s, and then plug them in and try again.
So to be perfectly clear – you must boot both controllers with the grub menu option of “Nimble Array Appliance Sanitization Boot”.
And sorry – but I didn’t actually get a picture of that screen, but here’s what it looks like just prior to the grub boot menu opening – as I mentioned Controller A was lagging about ½ second behind Controller B, so that is why the Putty session for COM4 is mostly empty – it populated and looked just like COM5 as I finished the screenshot.
After a minute or two, you’ll find yourself with two Putty serial console sessions full of yellow text. Eventually the output will show you the number of HDDs and SSDs detected in the system, along with an estimate of how long it will take to complete the sanitization (in my case, it predicted approximately 10 hours for 0 HDDs and 12 SSDs, but it really took about 8hr to complete based on the time stamps in my screenshots). Verify these drive count numbers match what you are expecting. If your drive count is what you expect it to be then continue on, otherwise you’ll need to troubleshoot why you aren’t seeing all the drives (perhaps you forgot to power on the expansion shelf first or something if you don’t see what you are expecting). It will also show you the Nimble serial number, the current time (in UTC), and be prompting you for the Sanitization Key.
Working in the Putty serial console session for Controller A (the serial console output refers to this as Controller 0), copy and paste the Sanitization Key into the session from the email you received from Nimble Support. Next you’ll be prompted for the type of sanitization you wish to perform. Your options are:
1.) All Drives and NVDIMM
2.) NVDIMM Only
3.) Drives Only
4.) Quit
**WARNING – There is no going back and no yes/no confirmation when you select one of these options, so make sure you are fully certain you are really ready to destroy this Nimble.**
Select option 1 and the secure wipe / sanitization process immediately starts.
You’ll first see the screen scroll with the NVRAM being zero’d out. This happens pretty much instantaneously. Next, the array will begin to write to all the drives. It does this in 4 different phases, and those phases differ depending on whether the drive is a HDD or SSD. For each hard drive, it writes all zeros to the drive, then it writes all ones to the drive, then it writes a random pattern, and then finally it verifies the random pattern. For each SSD, it writes a random pattern, then verifies the random pattern, then it writes another random pattern, which it then verifies again.
The only time your serial console will show activity is at the beginning of each new phase of writing / verifying, or when the sanitization is complete. This is why you don’t want to disconnect /close your Putty serial console sessions, because otherwise you will not be able to determine what phase you are in or if the process has completed.
Once the secure wipe completes on all the drives, you’ll see a pass/fail result for each drive, along with a statement that the Array Sanitization has completed.
Congratulations – you now have a new boat anchor (or a completely sanitized array if you want to look at it that way too)! You can now pull the power cables from the array and proceed to safely dispose of the asset in an environmental friendly way.
But what if you want to repurpose this array now? That blog post is for another day, but will be coming soon. Stay tuned!
Bits & Other Bytes 