HOWTO: Scheduled a standalone VMware ESXi Host Reboot via Powershell

We have several clients who have standalone VMware ESXi hosts (that are not part of any vCenter) without any option for vMotion or Storage vMotion.  This can make it difficult for us to keep those hosts current with patches, updates, and BIOS / firmware because it means we need to manually shut the hosts’ guest down, and then restart the host – none of which can be done during normal business hours – and I’m getting too old to work all night.

Fortunately, VMware provides us a way to use PowerShell to shutdown a ESXi host’s guest, and then force a reboot.  This means we can apply patches and updates late in the day to the ESXi host, then schedule the host to reboot early in the morning after the daily backup completes.  Then when we come into the office in the morning (usually an hour or two before the clients arrive at their offices), it is simply a matter of checking the host to ensure it is back up along with all it’s guests.

To schedule a standalone VMware Host reboot, the current VMware PowerCLI client needs to be installed on the machine that will be running the scheduled reboot.

Once the VMware PowerCLI is installed, you need to create 3 files:

  • C:\WINDOWS\VMWARE_ROOT.PWD – encrypted file that contains the root user’s password
  • C:\WINDOWS\VMWARE_HOST_REBOOT.CMD – the wrapper that will call PowerShell from TaskScheduler
  • C:\WINDOWS\VMWARE_HOST_REBOOT.PS1 – the actual PowerShell script that executes the reboot

To create the file C:\WINDOWS\VMWARE_ROOT.PWD, open PowerShell and run the following command:

read-host -assecurestring "Enter Password" | convertfrom-securestring | out-file C:\WINDOWS\VMWARE_ROOT.PWD

 

At the “Enter Password” prompt, enter the password of the root user account for the ESXi host you want to reboot.

You also need to set the PowerShell Execution Policy to support remote signed scripts such as C:\WINDOWS\VMWARE_HOST_REBOOT.PS1.  To do this, in PowerShell run the following command and select Yes when prompted:

Set-ExecutionPolicy RemoteSigned

We need to schedule a time for VMWARE_HOST_REBOOT.CMD to run.  I’ve set 4:15 am local time on March 22, 2015 in the example shown below, but you can adjust as required.  In an administrative command prompt, run this (***note – this will create the scheduled task to run as the currently logged in user***):

schtasks /create /tn "VMware Host Reboot" /tr C:\WINDOWS\VMWARE_HOST_REBOOT.CMD /sc once /st 04:15:00 /sd 03/22/2015 /rp "*" /ru "%userdomain%\%username%"

Now we need to create C:\WINDOWS\VMWARE_HOST_REBOOT.CMD, which is the batch file task scheduler uses to launch our PowerShell script.

rem --- begin cut and paste of notepad C:\WINDOWS\VMWARE_HOST_REBOOT.CMD
@echo off
C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -ExecutionPolicy RemoteSigned -noprofile -File C:\WINDOWS\VMWARE_HOST_REBOOT.PS1
exit /b
rem --- end cut and paste of C:\WINDOWS\VMWARE_HOST_REBOOT.CMD ---

Lastly, we need to create C:\WINDOWS\VMWARE_HOST_REBOOT.PS1, adjusting the variable for $server to the host you wish to reboot (all variables are all defined at the top of the script) and adjust wait time ($waittime) before force rebooting after you issue a graceful gust shutdown command.

###--- begin cut and paste of notepad C:\WINDOWS\VMWARE_HOST_REBOOT.PS1
### VMWARE_HOST_REBOOT.PS1
### @deancolpitts – http://blog.jbgeek.net
### 2015.03.20

### This script will attempt to perform a graceful VM restart via the VMware Tools inside the guest.
### Variables - please only adjust server, user, and waittime. Any other variables should not be touched.
### Server is the vCenter server or ESXi host's FQDN, while user is the vCenter user or ESXi user account.

$server = "VMWARE.FQDN.DOMAIN_OR_IPADDRESS"
$user = "root"
$waittime = "300"

$credentialFile = "C:\WINDOWS\VMWARE_ROOT.PWD"
$pass = cat $credentialFile | convertto-securestring
$credentials = new-object -typename System.Management.Automation.PSCredential -argumentlist $user,$pass

add-pssnapin VMware.VimAutomation.Core -ErrorAction SilentlyContinue -WarningAction SilentlyContinue | Out-Null
if ( $DefaultVIServers.Length -lt 1 )
{
Connect-VIServer -Server $server -Protocol https -credential $credentials -WarningAction SilentlyContinue | Out-Null
}

Get-VM | Shutdown-VMGuest -confirm:$false -WarningAction SilentlyContinue

### Wait x number of seconds for all the VM's to gracefully shutdown before a forced kill occurs
Start-Sleep -s $waittime

Restart-VMHost -VMHost $server -force -confirm:$false

###--- end cut and paste of C:\WINDOWS\VMWARE_HOST_REBOOT.PS1 ---

All that is left do now is wait for C:\WINDOWS\VMWARE_HOST_REBOOT.CMD to run at your scheduled time.

As always – Use any tips, tricks, or scripts I post at your own risk.

KB3148812 breaks Windows Server Update Services

Earlier this week, Microsoft pushed out KB3148812, which enables ESD decryption provisioning in WSUS (on Windows Server 2012 and Windows Server 2012 R2).  From what I read and understand, KB3148812 is going to be a mandatory update for WSUS to support Windows 10 updates after May 1.  Unfortunately, it appears that KB3148812 also breaks the WSUS console.  Rather than re-issue KB3148812 so it doesn’t break WSUS, Microsoft has published in a blog posting the necessary post-install steps to “un-break” WSUS after install KB3148812.

Basically you need to perform two steps to return WSUS to a working condition on Windows 2012.  First, you need to re-run the WSUS post-install.  And then you also need to add HTTP Activation to your WSUS server.

For Step 1, to re-run post-install, open an administrative command prompt and run:

"C:\Program Files\Update Services\Tools\Wsusutil.exe" postinstall /servicing

For Step 2, to install HTTP Activation, open an administrative PowerShell command prompt and run:

Install-WindowsFeature -Name NET-WCF-HTTP-Activation45

No reboot should be necessary and the WSUS console should now open and function normally for you.  There is the possibility you’ll still get client connectivity issue though with an error of 0x80244007, which is something entirely different that Microsoft is still looking at as of 2016.04.22.

Setup hourly HPE Insight Remote Support Service checking

In a previous post, I mentioned we utilize HPE Insight Remote Support (IRS) at all our client sites, and discovered the lovely undocumented “feature” that IRS has, which is a tendency not to start after a Windows server reboot after an IRS update. This great undocumented feature defeats the entire purpose of IRS – monitoring and alerting your HPE hardware. After getting burned by this feature three or four times in a month where customers noticed hardware faults (via amber alert lights on the equipment) before we did since IRS was not running to alert us, I decided it was time to write a script to check IRS hourly and alert us if it wasn’t running.

To configure Windows to send an alert if the HP IRS Service is stopped, create the following two files (file contents are at the end of this post) on the IRS server:

  • check_irs_service_status.cmd – which is the wrapper that will call PowerShell from Task Scheduler
  • check_irs_service_status.ps1 – which is the actual PowerShell script that executes the service status check

Lastly, we need to schedule check_irs_service_status.cmd to run hourly. I’ve set 2 minutes after the hour in the example shown below, but you can adjust as required.

schtasks /create /tn "Hourly IRS Service Check" /tr c:\Windows\check_irs_service_status.cmd /sc minute /mo 60 /st 00:02:00 /rp "*" /ru "%userdomain%\%username%"

By default, the SMTP from address will be the netbios computer name of the IRS server @ the User’s DNS Domain FQDN (i.e. IRS-SERVER@JBGEEK.NET).  The SMTP to address will be support @ the User’s DNS Domain FQDN (i.e. SUPPORT@JBGEEK.NET), and the SMTP server will be mail @ the User’s DNS Domain FQDN (i.e. MAIL.JBGEEK.NET).  You can determine what these will be by checking the system’s environment variables with SET from a command prompt.  You can customize these settings in the “Send-MailMessage” command if necessary.

All that is left to do is to stop the service and test run check_irs_service_status.cmd to verify the Send-MailMessage works properly in your environment.

 

check_irs_service_status.cmd

rem --- begin cut and paste of notepad c:\windows\check_irs_service_status.cmd
@echo off
C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -ExecutionPolicy RemoteSigned -noprofile -File C:\Windows\check_irs_service_status.ps1
exit /b
rem --- end cut and paste of c:\windows\check_irs_service_status.cmd ---

 

check_irs_service_status.ps1

###--- begin cut and paste of notepad c:\windows\check_irs_service_status.ps1
### Check_irs_service_status.ps1
### @deancolpitts – http://blog.jbgeek.net
### 2016.01.27
### This script will check the status of the server HPRSMAIN and alert via email if the service is stopped.

$Service = Get-Service -name HPRSMAIN
$Service.Status
if ($Service.Status -eq "Stopped") {
 $CurrentTime = Get-Date
 Send-MailMessage -From "$env:computername@$env:userdnsdomain" -To "support@$env:userdnsdomain" -Subject "$env:computername - HP IRS Service is stopped!!!" -Body "The HP IRS Service is stopped on $env:computername.$env:userdnsdomain at approximately $CurrentTime." -Priority High -DNO onSuccess, onFailure -SmtpServer "mail.$env:userdnsdomain"
}

###--- end cut and paste of notepad c:\windows\check_irs_service_status.ps1

 

HOWTO: Schedule Daily Netscaler VPX Reboots via Powershell

We often utilize Citrix’s NetScaler VPX running on VMware ESXi 5.5 to allow our clients to securely connect to their Citrix infrastructure from outside the firewall.  For the most part – it works well.  Unfortunately though, our experience has taught us that occasionally NSVPX goes all fubar on it’s own after a few days of running and stops processing connection requests once the user logs in.  A simple reboot of the NSVPX VM usually resolves the user’s connectivity issues..

To combat this issue, I wrote a Powershell script that we run as a daily scheduled task on our management server to have vCenter automatically restart the machine once a day.  You could easily modify this script to reboot any VM you want though.

To configure daily VM rebooting, the current VMware PowerCLI client needs to be installed on the machine that will be running the scheduled reboot.  Once the VMware PowerCLI is installed, you need to create 3 files on the management machine:

  1. daily_nsvpx_reboot.cmd – which is the wrapper that will call PowerShell from TaskScheduler (see below in for cut and paste of the file contents)
  2. daily_nsvpx_reboot.ps1 – which is the actual PowerShell script that executes the reboot (see below for cut and paste of the file contents)
  3. daily_nsvpx_reboot.pwd – which is an encrypted file that contains the vCenter user’s password

To create the file daily_nsvpx_reboot.pwd, open PowerShell and run the following command:

read-host -assecurestring "Enter Password" | convertfrom-securestring | out-file c:\windows\daily_nsvpx_reboot.pwd

At the “Enter Password” prompt, enter the password of the user account you will be using that has rights in vCenter or the ESXi host to perform VM restarts.

You may also need to set the PowerShell Execution Policy to support remote signed scripts such as daily_nsvpx_reboot.ps1.  To do this, open PowerShell and run the following command and select Yes when prompted:

Set-ExecutionPolicy RemoteSigned

After creating daily_nsvpx_reboot.cmd and daily_nsvpx_reboot.ps1 (see below for file contents of these two files), edit daily_nsvpx_reboot.ps1 and adjust the variables for $server, $user, and $vm2reboot to fit your environment (these three variables are all defined at the top of the script).

Lastly, you need to schedule daily_nsvpx_reboot.cmd to run daily.  I’ve set 4:15 am local time in the example shown below, but you can adjust as required.  To schedule the task, open an Administrative command prompt and run the following command (adjust domain\username to be the same user account that has rights in vCenter or the ESXi host to perform VM restarts):

schtasks /create /tn "Daily NSVPX Reboot" /tr C:\WINDOWS\DAILY_NSVPX_REBOOT.CMD /sc daily /st 04:15:00 /rp "*" /ru "domain\username"

All that is left do now is test run daily_nsvpx_reboot.cmd and see that it runs and reboots the NSVPX.  If you are monitoring via ProcExp or TaskManager on the management machine, you should note low CPU usage followed by several spikes up to 50% (it is single threaded), and you should be able to see in the NSVPX console via vCenter when it reboots.

And as always – Use any tips, tricks, or scripts I post at your own risk.


daily_nsvpx_reboot.cmd – file contents

rem — begin cut and paste of notepad c:\windows\daily_nsvpx_reboot.cmd
@echo off
C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -ExecutionPolicy RemoteSigned -noprofile -File C:\Windows\daily_nsvpx_reboot.ps1
exit /b
rem — end cut and paste of c:\windows\daily_nsvpx_reboot.cmd —

daily_nsvpx_reboot.ps1 – file contents

    ###— begin cut and paste of notepad c:\windows\daily_nsvpx_reboot.ps1

    ### Daily_nsvpx_reboot.ps1
    ### @deancolpitts – http://blog.jbgeek.net
    ### 2015.01.02
    ### This script will attempt to perform a graceful VM restart via the VMware Tools inside the guest.

    ### Variables – please only adjust server, user, and vm2reboot.  Any other variables should not be touched.
    ### Server is the vCenter server or ESXi host’s FQDN, while user is the vCenter user or ESXi user account.
    ### if any smtp variables present, they should be self-explanatory.

    $server = “vcenter.domain.fqdn”
    $user = “vcenter_username”
    $vm2reboot = “nsvpx”

    ### Read the encrypted user password from “c:\windows\daily_nsvpx_reboot.pwd”
    ### Use the following commented out PowerShell command to manually create a new credentials store.
    ### Enter the user’s password when prompted while running the read-host command
    ### read-host -assecurestring “Enter Password” | convertfrom-securestring | out-file c:\windows\daily_nsvpx_reboot.pwd

    $credentialFile = “c:\windows\daily_nsvpx_reboot.pwd”
    $pass = cat $credentialFile | convertto-securestring
    $credentials = new-object -typename System.Management.Automation.PSCredential -argumentlist $user,$pass

    add-pssnapin VMware.VimAutomation.Core -ErrorAction SilentlyContinue -WarningAction SilentlyContinue | Out-Null

    if ( $DefaultVIServers.Length -lt 1 )
    {
    Connect-VIServer -Server $server -Protocol https -credential $credentials -WarningAction SilentlyContinue | Out-Null
    }

    Restart-VM -VM $vm2reboot -RunAsync -Confirm:$false

    ###— end cut and paste of c:\windows\daily_nsvpx_reboot.ps1 —