HPE

Upgrade a stuck ILO firmware via SSH

@DeanColpitts3 Comments

We have had a rash of issues where by upgrading ILO firmware via the WebUI has been failing.  It looks like it finishes, but when you log back in, it is still the original firmware from when you started the upgraded.  And no matter what you do via the WebUI, it just will not upgrade.  So to upgrade the stubborn firmware, the simplest thing to do is SSH the ILO directly and upload the firmware via the console interface.  Below are the steps to do this.

First, you need a running web server to pull the firmware from.  IIS is usually the handiest, so it is simply a matter of adding a mime-type for the binary firmware file.  Open an administrative command prompt and run:

c:\windows\system32\inetsrv\appcmd.exe set config /section:staticContent /+"[fileExtension='.bin',mimeType='application/x-bin']"
iisreset /restart

Extract the ILO firmware bin with 7-Zip and put the bin somewhere within IIS that you can download it.   Next – to save myself extra grief, I also make sure I can actually download the firmware to a regular PC with a browser before continuing.  So open the browser of your choice and make sure you can download the bin to your PC before continuing.

Putty the ILO interface, accepting the SSH key (if prompted), and login.  Once logged in, check, then download the new firmware with the following commands.

*** Note – the ILO will automatically reboot once it successfully downloads the firmware and does not give any indication of the reboot.  As a result, you might want to start a continuous ping to the ILO to see once it has rebooted and is back up ***

show /map1/firmware1
cd /map1/firmware1
load -source http://http_server_ip/ilox_xxx.bin

Once the ILO reboots, you should have a working ILO with the firmware version you want / need.

As always – Use any tips, tricks, or scripts I post at your own risk.

2016.05.12 - 19.34.52 - SNAGIT - 0097

Factory Reset a HPE FlexFabric 5700 to defaults

@DeanColpittsLeave a comment

Not to long ago, we received a new HPE FlexFabric 5700 switch and we proceeded to muck around with the configuration settings trying a few things that we normally would never do to a production switch.  When we were done having fun and learning, we needed to reset the unit back to defaults so we could really deploy it into production.  Of course, resetting a switch to factory defaults is not something you do very often, so we had to actually RTFM.  I’ll save you the time of that here…

From the serial console, execute these commands:

restore factory-default
yes
save
yes
{hit enter}
reboot

When the switch reboots, it will be at defaults.

Below is a screen snapshot of what you’ll see during this process.

2016.05.11 - 14.43.21 - SNAGIT - 0066

HOWTO: Monitor the rebuild status of a HPE SmartArray in ESXi 5.5

@DeanColpittsLeave a comment

To monitor the rebuild status of a HP SmartArray controller in VMware ESXi 5.5, you need to have the HP VMware tools bundle installed (which is installed if the server was installed from the HP VMware media / ISO).  Once the tools bundle has been installed, simply SSH the server (or go right on the console, either physically or via ILO), login and run:

/opt/hp/hpssacli/bin/hpssacli ctrl all show status

This will provide you a list of all the SmartArray controllers in the server.  From this list, find the slow number of the controller that contains the logical drive you need to check the status on and run the following command (substitute slot=XX for the slot value you determined with the previous command):

/opt/hp/hpssacli/bin/hpssacli ctrl slot=XX ld all show

2016.04.14 - 09.12.11 - SNAGIT - 0000

If you happen to running an older version of ESXi 5.x, or your HP VMware Tools bundle is not somewhat recent, then the commands are somewhat different.  In this case the correct commands are:

/opt/hp/hpacucli/bin/hpacucli
ctrl all show
ctrl slot=0 ld all show

HPE Insight Remote Support 7.6 auto-upgrade fails

@DeanColpitts2 Comments

As some of you may have noticed, HPE rehpe_pri_grn_pos_rgbleased Insight Remote Support (IRS) version 7.6 this week.  Among other things, the interface is now rebranded with the new HPE logo and icon, it has better security logging, and add support for a bunch of new HPE Networking and HPE StoreEasy products.

If you have already set the “Automatic Update Level” in IRS to “Automatically Download and Install”, you may already have 7.6 successfully deployed to your server.  It’ll be pretty obvious to tell too – if you see the HPE logo shown above on the login page or as the desktop shortcut icon, you are already at version 7.6.

For some reason however, a couple of my IRS 7.5 servers have failed to auto-update to 7.6.  Trying to install the 7.6 update from the Software Tab in IRS by clicking the Start Update also fails.  Normally at this point, I’d simply go to the Software Depot, download 7.6 and manually run the setup – except that 7.6 isn’t available in the Software Depot as the Software Depot download page generates an error message as of this writing (2016.04.02).

So – after some troubleshooting and poking around the log files, I determined you can download the 7.6 package update from the same spot that IRS downloads it:

https://services.isee.hp.com/SWM/packages/ProdUpgPkg/2016-03-31T154720/ProdUpgPkg+7.6.0.27.zip

Unzip this archive to C:\TEMP and then from a command prompt run:

msiexec /i "C:\TEMP\ProdUpgPkg+7.6.0.27\lib\hprs7kit.msi" /lv "%HP_RS_LOG%\hprs_7.6.0_install.log"

Now – if your servers were like those same servers I have, this will fail too.  Taking a look at “%HP_RS_LOG%\hprs_7.6.0_install.log“, you’ll find that pg_dumpall.exe couldn’t connect to the database as the connection was refused.  This results in database.sql being missing, which causes the install to puke with an error code of 1603.  database.sql is the Postgres database dump of your production IRS database that the installer attempts to make.  Now just above the 1603 error in “%HP_RS_LOG%\hprs_7.6.0_install.log“, you’ll find the actual command line for pg_dumpall.exe, which should be (depending on the vintage of your original IRS install) either:

"C:\Program Files\HP\RS\postgresql_9_win32\bin\pg_dumpall.exe" --host=localhost --port=7950 --username=postgres --file="C:\ProgramData\HP\RS\DATA\PERSISTENCE\UPGRADE\database.sql"
-- or --
"C:\Program Files (x86)\HP\RS\postgresql_9_win32\bin\pg_dumpall.exe" --host=localhost --port=7950 --username=postgres --file="C:\ProgramData\HP\RS\DATA\PERSISTENCE\UPGRADE\database.sql"

Manually running the appropriate version command line from above will result in you being prompted for the postgres user password 6 times.  Unfortunately, this password is undocumented, but by doing some detective work (I won’t be sharing how I found what it was), I’ve determined it to be “edit – removed 2016.04.05 as per a request from HPE“.  So enter this password when prompted each of those 6 times, and you’ll find C:\ProgramData\HP\RS\DATA\PERSISTENCE\UPGRADE\database.sql is created.  Now you can go back and run the installer again from the command prompt:

msiexec /i "C:\TEMP\ProdUpgPkg+7.6.0.27\lib\hprs7kit.msi" /lv "%HP_RS_LOG%\hprs_7.6.0_install.log"

Your upgrade should now complete successfully, and all that is left is to log into IRS, go to the Software Tab and check for updates, and install any remaining updates.

As always – Use any tips, tricks, or scripts I post at your own risk.

 

Setup hourly HPE Insight Remote Support Service checking

@DeanColpittsLeave a comment

In a previous post, I mentioned we utilize HPE Insight Remote Support (IRS) at all our client sites, and discovered the lovely undocumented “feature” that IRS has, which is a tendency not to start after a Windows server reboot after an IRS update. This great undocumented feature defeats the entire purpose of IRS – monitoring and alerting your HPE hardware. After getting burned by this feature three or four times in a month where customers noticed hardware faults (via amber alert lights on the equipment) before we did since IRS was not running to alert us, I decided it was time to write a script to check IRS hourly and alert us if it wasn’t running.

To configure Windows to send an alert if the HP IRS Service is stopped, create the following two files (file contents are at the end of this post) on the IRS server:

  • check_irs_service_status.cmd – which is the wrapper that will call PowerShell from Task Scheduler
  • check_irs_service_status.ps1 – which is the actual PowerShell script that executes the service status check

Lastly, we need to schedule check_irs_service_status.cmd to run hourly. I’ve set 2 minutes after the hour in the example shown below, but you can adjust as required.

schtasks /create /tn "Hourly IRS Service Check" /tr c:\Windows\check_irs_service_status.cmd /sc minute /mo 60 /st 00:02:00 /rp "*" /ru "%userdomain%\%username%"

By default, the SMTP from address will be the netbios computer name of the IRS server @ the User’s DNS Domain FQDN (i.e. IRS-SERVER@JBGEEK.NET).  The SMTP to address will be support @ the User’s DNS Domain FQDN (i.e. SUPPORT@JBGEEK.NET), and the SMTP server will be mail @ the User’s DNS Domain FQDN (i.e. MAIL.JBGEEK.NET).  You can determine what these will be by checking the system’s environment variables with SET from a command prompt.  You can customize these settings in the “Send-MailMessage” command if necessary.

All that is left to do is to stop the service and test run check_irs_service_status.cmd to verify the Send-MailMessage works properly in your environment.

 

check_irs_service_status.cmd

rem --- begin cut and paste of notepad c:\windows\check_irs_service_status.cmd
@echo off
C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -ExecutionPolicy RemoteSigned -noprofile -File C:\Windows\check_irs_service_status.ps1
exit /b
rem --- end cut and paste of c:\windows\check_irs_service_status.cmd ---

 

check_irs_service_status.ps1

###--- begin cut and paste of notepad c:\windows\check_irs_service_status.ps1
### Check_irs_service_status.ps1
### @deancolpitts – http://blog.jbgeek.net
### 2016.01.27
### This script will check the status of the server HPRSMAIN and alert via email if the service is stopped.

$Service = Get-Service -name HPRSMAIN
$Service.Status
if ($Service.Status -eq "Stopped") {
 $CurrentTime = Get-Date
 Send-MailMessage -From "$env:computername@$env:userdnsdomain" -To "support@$env:userdnsdomain" -Subject "$env:computername - HP IRS Service is stopped!!!" -Body "The HP IRS Service is stopped on $env:computername.$env:userdnsdomain at approximately $CurrentTime." -Priority High -DNO onSuccess, onFailure -SmtpServer "mail.$env:userdnsdomain"
}

###--- end cut and paste of notepad c:\windows\check_irs_service_status.ps1

 

HPE Insight Remote Support fails to start after reboot

@DeanColpittsLeave a comment

We utilize HPE Insight Remote Support (IRS) at all our client sites, and typically have it running on either Windows 2008 R2 or Windows 2012 R2.  To simplify administration, we typically enable auto-update of IRS, which means IRS will download updates from HPE as they become available and self-update.  One of the lovely “features” that we discovered is that upon the next Windows server reboot after an IRS update (typically at 3am on the first Wednesday after the 2nd Tuesday of every month – thanks Microsoft), the HPRSMain service fails to start.  No amount poking, prodding or swearing will convince the service to start either.

The solution is to run a repair – except the HPE team doesn’t make that easy either as the only option in Add/Remove programs is to uninstall.  Fortunately, you should find the .msi for IRS in C:\ProgramData\HP\RS\DATA\SWM\LANDINGZONE\ProdUpgPkg\unzipped\lib.

So the quickest way to fix IRS at this point is to open an Administrative Command Prompt and run:

msiexec /f "C:\ProgramData\HP\RS\DATA\SWM\LANDINGZONE\ProdUpgPkg\unzipped\lib\hprs7kit.msi" /lv "%HP_RS_LOG%\hprs_recovery.log"

After a few minutes, the HPRSMain service should start and good until at least the next IRS update.

HOWTO: Make a 3Par RJ45 serial console cable for an OpenGear ACM5004 Remote Site Manager

@DeanColpitts2 Comments

On all the 3Par StoreServ 7000/8000 nodes we manage for our clients, we utilize OpenGear ACM5004-F-E Remote Site Manager (basically a serial console switch) so that we can remotely access the serial console on the nodes (in case a node is shutdown to the SMI Whack prompt or something else silly is going on).  The ACM5004 has 4 x RJ45 serial ports on it, which are a Cisco Straight pinout.

Just plugging a standard patch cable into the ACM5004 and the back on the StoreServ node isn’t going to work.  And using the RJ45/DB9 couplers that the 3Par ships with is both a pain in the butt, an eye sore, and bulky.  So I set out to build my own custom RJ45 cable.

On the 3Par end, use pins 2 (orange), pin 3 (blue), and pin 5 (green). Be sure to label this end as the 3Par end!!! I attempt to always put a small tiewrap (which is white in the photo below and hard to see thanks to a blurry photo) around this end of the cable to mark it as the 3Par 3nd.

On the ACM5004 end, use pins 3 (blue), pin 5 (green), and pin 6 (orange). Be sure to label this end as ACM5004!!!

Plug the 3Par end into the node’s serial console port, and the other end into the ACM5004.  After configuring the ACM5004, SSH it and connect the appropriate port on the ACM5004 and you should now see your 3Par serial console.

3Par
3Par End
ACM5004
ACM5004 End