HOWTO: Using #PowerShell to ensure a #Veeam USB Repository always has the correct drive letter

Some time ago I had a customer who switched completely to Veeam from Backup Exec (yeah baby!!!).  Offsite replication of any sort was out of the question as the customer simply couldn’t get the necessary bandwidth from any of the ISPs that serviced the area at anything that even approached an unreasonable price.  Instead we opted for a HPE StoreOnce for onsite and some sort of removable drive system for daily offsite.  The customer insisted on using Western Digital My Books (USB3) due to their capacity (6TB) and semi-ruggedness for the offline repo.  The drives are rotated out each day by a staff member for a new drive and they send the previous night’s drive offsite.

The My Books were originally setup using F: as the drive letter.  This would work great until someone plugged a USB key into the server and it was auto-assigned F: by Windows 2012R2, or if someone logged in who had a drive mapping of F:.  And then other days, Windows would randomly assign a drive letter other than F: even though F: was available.  At this point, Veeam would puke because it couldn’t find the repo anymore.  And occasionally the customer would forget to swap the drive for a fresh one (or even correct drive some days), which meant there wouldn’t be enough free space on the repo to complete that night’s backup (about 4.9TB is processed daily).

To work around all this, I ended up writing a PowerShell script to always map the My Book to Z:, clean the previous backups off of it (if any were found, or if the wrong drive with a different backup set was plugged in), and start the actual backup job.  And if the script can’t find the My Book or free disk capacity is less than 95% after attempting cleanup, it will abort the backup and send an email warning to the support team that the backup was aborted.  Rather than have Veeam schedule the job to run, I use Windows Task Scheduler instead to run the script, then use the Veeam PowerShell module to start the backup job from inside the script after I’ve verified Z: is present.

This script assumes that every rotated My Book drive has been formatted and has a volume label containing “WD MY BOOK”.  It also assumes the SMTP server is called mail dot whatever the DNS domain name of the machine running the script is (i.e. my AD is jbgeek.net, so mail.jbgeek.net).  All of my client sites have a cname for their Exchange server called “mail” created in their AD DNS zone – which means I can cut and paste the same script without modification from one client site to another, which cuts down the chance of an editing error, and makes it quicker to deploy new scripts when necessary for my team.

You will need to adjust the “Get-ChildItem -Path” (lines 21 to 26) directories to fit your environment, along with the $SendEmailTo variable (line 7) and the “Start-VBRJob -Job” job name (line 41).  Line 28 defines the cutoff point for disk capacity – if the available disk space is less than 95% of the drive capacity, the backup aborts.  Line 1 is commented it out – I generally always start my scripts with # start notepad++ script.filename so that I can just cut and paste it into a command prompt to create my script file – that way as I on-board each new customer, their sites are configured the same as existing sites.  Again, it cuts down the chances of an error and makes it quicker for my guys to deploy new scripts.

**NOTE – the following deletes data from your backup cartridges. Use any tips, tricks, or scripts I post at your own risk.  I accept zero liability and responsibility if you use these scripts!!!**

Here is the command line to create the scheduled task to run as the logged in user at 10pm each weekday (it will prompt you for your password when you run it in a command prompt):

schtasks /create /tn "Weekday Veeam USB Drive Backup" /tr "\"C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe\" -ExecutionPolicy RemoteSigned -noprofile -File  C:\Windows\Run_External_Drive_Veeam_Backup.ps1" /sc daily /st 22:00:00  /rp "*" /ru "%userdomain%\%username%"

Here is the Powershell script (updated 2017.01.17 to fix an error):

# start notepad++ C:\Windows\Run_External_Drive_Veeam_Backup.ps1

add-pssnapin Veeam*

$DnsDomain = Get-WmiObject -Class Win32_NTDomain -Filter "DSDirectoryServiceFlag='True'" | Select -ExpandProperty DnsForestName
$ThisComputerName = Get-WmiObject -Class Win32_ComputerSystem | Select -ExpandProperty Name
$SupportEmailAddress = "support@$DnsDomain"

# Get-WmiObject Win32_Volume | Where-Object {$_.DriveLetter -eq $null}
$drives = Get-WmiObject -Class win32_volume | Where-Object {$_.Label -match "WD MY BOOK"}
$i = 0
Foreach($drive in $drives)
{
#Set letter
$DriveLetter = "Z:"
Set-WmiInstance -input $drive -Arguments @{DriveLetter="$DriveLetter"}
$i++
}

if (Test-Path -path Z:\){
Get-ChildItem -Path "Z:\Veeam\Backups\All VMs - Z Drive" -Include *.vbm -recurse | foreach { $_.Delete()}
Get-ChildItem -Path "Z:\Veeam\Backups\All VMs - Z Drive" -Include *.vib -recurse | foreach { $_.Delete()}
Get-ChildItem -Path "Z:\Veeam\Backups\All VMs - Z Drive" -Include *.vbk -recurse | foreach { $_.Delete()}
Get-ChildItem -Path "Z:\Veeam\Backups\All VMs" -Include *.vbm -recurse | foreach { $_.Delete()}
Get-ChildItem -Path "Z:\Veeam\Backups\All VMs" -Include *.vib -recurse | foreach { $_.Delete()}
Get-ChildItem -Path "Z:\Veeam\Backups\All VMs" -Include *.vbk -recurse | foreach { $_.Delete()}
$externaldrivelabel = Get-WmiObject -Class win32_volume | Where-Object {$_.Label -match "WD MY BOOK"} | Select -ExpandProperty label
Get-WmiObject -Class win32_volume | Where-Object {$_.Label -match "WD MY BOOK"} | % { if (($_.FreeSpace/$_.Capacity) -le '0.95' )
{Send-MailMessage -From "$($ThisComputerName.ToUpper())@$($DnsDomain.ToUpper())" -To "$SupportEmailAddress" `
-Subject "Backup Aborted due to low space on $($externaldrivelabel)." `
-Body  "Aborting nightly Veeam backup to $($externaldrivelabel) (Z:) on $($ThisComputerName.ToUpper()).$($DnsDomain.ToUpper()) due to low disk space." `
-Priority High -DNO onSuccess, onFailure -SmtpServer "mail.$DnsDomain"
break}
}
}
if (Test-Path -path Z:\){
Send-MailMessage -From "$($ThisComputerName.ToUpper())@$($DnsDomain.ToUpper())" -To "$SupportEmailAddress" `
-Subject "Now starting nightly Veeam backup to $($externaldrivelabel)." `
-Body  "Now starting nightly Veeam backup to $($externaldrivelabel) (Z:) on $($ThisComputerName.ToUpper()).$($DnsDomain.ToUpper())." `
-Priority High -DNO onSuccess, onFailure -SmtpServer "mail.$DnsDomain"
Start-VBRJob -Job "All VMs - Z Drive" -FullBackup -RunAsync
break
}else{
Send-MailMessage -From "$($ThisComputerName.ToUpper())@$($DnsDomain.ToUpper())" -To "$SupportEmailAddress" `
-Subject "Error:  No external backup drive (Z:) found on $($ThisComputerName.ToUpper()).$($DnsDomain.ToUpper())!!!" `
-Body  "Error:  Cannot find or mount the external backup drive (Z:) on $($ThisComputerName.ToUpper()).$($DnsDomain.ToUpper())!!!  Now aborting nightly backup to external hard drive!!!" `
-Priority High -DNO onSuccess, onFailure -SmtpServer "mail.$DnsDomain"
break
}

As always – Use any tips, tricks, or scripts I post at your own risk.

HOWTO: Converting from BackupExec to #Veeam when using RDX drives

Ok – I’m completely done with Backup Exec when it comes to VMware.  I’ve been selling, supporting, certified on and even using Backup Exec for our own internal backups since it was Conner Backup Exec for Windows NT 3.1, way back in 1993.  Once upon a time, it was a great product – in fact it was the only product for backups that worked worth a damn.  But it’s reliability has dropped to nothing over the past 6 or 7 years.  Technical support has been off-shored and 99.9% of the time, if I am lucky enough to finally reach someone in technical support on the phone, I can’t understand a damn word they say due to their thick accent and shitty VOIP lines crossing the Pacific Ocean.  Today was the last straw with Backup Exec, their crappy bugs, and unreliable VMware backups.  So now it’s time to fully embrace the move to Veeam, which I’ve been considering for some time (note of disclosure – I am also a certified Veeam VMCE – v7, v8, & v9)

Several of my clients have single standalone ESXi hosts, an HPE StoreOnce appliance, a physical Windows Server 2012R2 with a RDX drive or two (for offline backups), and both Backup Exec and Veeam loaded on that Windows server.  Oh – and many, many, many RDX cartridges that have months of rotated backups on them that are all three quarters full.  I can’t just erase all these cartridges in one swoop and use them for Veeam backups.  And I certainly don’t want to have to log into the clients’ servers everyday to manually delete the old Backup Exec folders off the RDX (as they come up in rotation) so that there is enough room for the nightly Veeam backup.  And finally, even though I’m dumping Backup Exec for my VMware backups, I still need to use Backup Exec to backup the 2012R2 physical instance to the same RDX cartridge that Veeam is going to use (atleast until Veeam releases their next project).  So what do I do?

A little PowerShell scripting to the rescue – that is what I am going do!

After going through a sampling of several RDX cartridges at several different client sites, I’ve determined that when Backup Exec runs with GRT enabled it dumps those backed up VMs in IMGxxxxxx folders on the root of the RDX drive (including the VMDKs).  I also discovered (or at least in the environments that I’ve setup) that GRT enabled application backups (not VMs, but rather SQL, AD, Exchange) will also be in an IMG folder with either a file called ntds.dit or edb.chk, and sometimes both!  In my case, my 2012R2 server has SQL and AD on it, so I want to be careful not to delete IMG folders that potentially contain my SQL and AD backups (which could screw Backup Exec up even more than normal when it uses that cartridge again for the 2012R2 server).

In the end, I setup the RDX drive as a new rotated drive repository in Veeam (prior to this Veeam only backed up to the HPE StoreOnce).  I then create a new Veeam job that did active fulls to the RDX drive every night (with a restore points to keep of 1).  In the job’s Advanced Settings menu, I added a pre-run script that runs C:\Windows\Remove_BackupExec_IMG_Folders.cmd.  This script in turn launches a PowerShell script that deletes all the IMGxxxxxx folders off the RDX drive except IMG folders that contain either ntds.dit or edb.chk.

**NOTE – the following deletes data from your backup cartridges. Use any tips, tricks, or scripts I post at your own risk.  I accept zero liability and responsibility if you use these scripts!!!**

Here is the contents of my batch file.

rem start notepad++ "C:\Windows\Remove_BackupExec_IMG_Folders.cmd"
PowerShell -NoProfile -ExecutionPolicy Bypass -Command "& {Start-Process PowerShell -ArgumentList '-NoProfile -ExecutionPolicy Bypass -File ""C:\Windows\Remove_BackupExec_IMG_Folders.ps1""' -Verb RunAs}"
exit/b

Here is the contents of the PowerShell script to remove the IMGxxxxxx folders (adjust the drive letter accordingly)

# start notepad++ "C:\Windows\Remove_BackupExec_IMG_Folders.ps1"
foreach ($i in Get-ChildItem R:\IMG*)
{if ((test-path "$i\ntds.dit") -eq $False -and (test-path "$i\edb.chk") -eq $False) {Remove-Item $i -force -recurse -confirm:$false}}

But wait! There is more!

Because I am still going to have to suffer with Backup Exec a while longer to backup my 2012R2 server, I need to make sure my nightly Backup Exec job doesn’t eject the RDX cartridge on me before Veeam finishes it’s RDX job.  To ensure this, I disabled the scheduled RDX jobs on my Backup Exec server.  Fortunately, Backup Exec includes a PowerShell module called BEMCLI.  So I wrote a second set of scripts as it was simply a matter of starting PowerShell from a script, importing the module, and starting the job.  So this time my scripts are a post-job script to start the Backup Exec job only after the Veeam job completes.

2017-01-11-16-33-43-snagit-0002

Here is the batch file to launch PowerShell.

rem start notepad++ "C:\Windows\Start_BE_UTIL01_RDX_JOB.cmd"
PowerShell -NoProfile -ExecutionPolicy Bypass -Command "& {Start-Process PowerShell -ArgumentList '-NoProfile -ExecutionPolicy Bypass -File ""C:\Windows\Start_BE_UTIL01_RDX_JOB.ps1""' -Verb RunAs}"
exit/b

And here is the PowerShell script to start the Backup Exec job called “23:10 UTIL01 RDX-Full”.

# start notepad++ "C:\Windows\Start_BE_UTIL01_RDX_JOB.ps1"
Import-Module BEMCLI
Get-BEJob -Name "23:10 UTIL01 RDX-Full" | Start-BEJob -confirm:$false

Now when my Veeam backup job to RDX starts, it deletes all the IMGxxxxxx folders off the RDX drive (unless those folders contain either ntds.dit or edb.chk), and when it completes, it starts the remaining Backup Exec job, which ultimately ejects the RDX cartridge when it completes.

As always – Use any tips, tricks, or scripts I post at your own risk.

HOWTO: Change a Windows’ network type from Public to Private via PowerShell

Stupid Windows Server 2016…  Stupid Windows 10…  Damn Microsoft…

Ever find yourself with a network interface that is stuck as Public and no obvious way to change it via GUI?

2016-12-24-14-24-28-snagit-0024

Five simple PowerShell lines to the rescue!

Get-NetConnectionProfile
$Profile = Get-NetConnectionProfile -InterfaceAlias "interface_alias_name"
$Profile.NetworkCategory = "Private"
Set-NetConnectionProfile -InputObject $Profile
Get-NetConnectionProfile

2016-12-24-14-25-51-snagit-0025

2016-12-24-14-26-07-snagit-0026

And now your network type should be set to Private network.

As always – Use any tips, tricks, or scripts I post at your own risk.

Blackberry BES 12 – Device Last Contact Reporting

We use Blackberry’s BES12 to manage all our mobile devices and all our clients’ mobile devices – be it Blackberry, Apple, Android, or even (shudder…) Windows Phone.  Blackberry BES12 is a fairly solid product, but with one major flaw that Blackberry still hasn’t fixed to my satisfaction.  Occasionally (well – more than occasionally), devices just plain old stop checking in and calling home to the BES. When this happens, the devices become unmanageable. At this point we usually need to contact the end user to have them manually open the BES12 client on their device, which will try to call home and reestablish connectivity.

Newer versions of Blackberry BES12 make it easy to show the last device contact time in the Advanced User view if you go in and turn it on in the GUI.  But unfortunately, BES12 doesn’t provide any way that I know of to get a scheduled report by email with these details. So this means we need to manually go log into customers’ BES servers on a regular basis to see if any devices have lost connectivity. This is a huge time sink and a pain in the rear for our help desk that we could probably do without.

So after some poking through the BES 12.5.1 database tables, I came up with a query that will give me the information I am looking for.  All that was left after this was to create a PowerShell wrapper to automatically run the script and email me the report on a daily basis (of which there is a picture below).

2016.08.30 - 09.39.16 - SNAGIT -  0013

Notes:

SQL Server 2012 Express or newer is required for PowerShell integration I believe. I’ve tested this script with SQL Express 2012 SP3. You can check for PowerShell integration support by using these two PowerShell commands:   Import-Module “SQLPS”   and   “Invoke-Sqlcmd”

I wrote the SQL query based on the SQL tables in BES 12.5.1, but it appears to work against BES 12.3.1 too.

Code lines 1 and 2 are comments I like to add to my scripts to allow me a quick way to copy, paste, and setup my script from Outlook onto the target machine(s) so everyone is setup the same way. Code line 2 will set the script to run at 12:15 am daily under the System security context. Adjust as you see fit.

Code lines 10, 11 and 26 are variables that need to be changed and adjusted for customer environments as required.

Code line 10 – $emailTo is who the email needs sent to (i.e. besadmin@jbgeek.net)
Code line 11 – $smtpServer is the SMTP server you are going to send through (i.e. mail.jbgeek.net)
Code line 26 – BES12 is the name of your production BES12 SQL database

By default, if you don’t change anything, $emailTo and $smtpServer will auto-populate using the $DnsDomain variable, just like $emailFrom, which should end up as the server’s NetBios computername @ the server’s DNS name (i.e. JBGEEK-BES01@JBGEEK.NET).

As always – Use any tips, tricks, or scripts I post at your own risk.

###  begin cut & paste of start notepad++ c:\windows\BES_Connectivity_Report.ps1
###  cmd.exe /c schtasks /create /tn "Daily BES Connectivity Report" /tr "\"C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe\" -ExecutionPolicy RemoteSigned -noprofile -File C:\Windows\BES_Connectivity_Report.ps1" /sc daily /st 00:15:00 /ru "System"
###  Dean Colpitts / blog.jbgeek.net / 2016.08.30

Import-Module 'SQLPS' -DisableNameChecking;

$DnsDomain = Get-WmiObject -Class Win32_NTDomain -Filter "DSDirectoryServiceFlag='True'" | Select -ExpandProperty DnsForestName
$ThisComputerName = Get-WmiObject -Class Win32_ComputerSystem | Select -ExpandProperty Name
$emailFrom = "$($ThisComputerName.ToUpper())@$($DnsDomain.ToUpper())"
$emailTo = "besadmin@$DnsDomain"
$smtpServer = "mail.$DnsDomain"
$messageSubject = "$ThisComputerName.$DnsDomain BES Connectivity Report"
$message = New-Object System.Net.Mail.MailMessage $emailfrom, $emailto
$message.Subject = $messageSubject
$message.IsBodyHTML = $true

$style = @'
<style type="text/css">
  table {text-align: left; font-family: arial, font-size: 12px; padding: 5px 5px; border: 1px solid #000000; border-collapse: collapse;padding-right: 10px; padding-left: 10px;}
  th {text-align: left; font-family: arial, font-size: 12px; padding: 5px 5px; color: #000; column-width: 100px; border-top: 1px solid #000000; border-bottom: 1px solid #000000;background-color: #6495ED;padding-right: 10px; padding-left: 10px;}
  td {font-family: arial, font-size: 12px; padding: 5px 5px; color: #000; column-width: 100px; border-top: 1px solid #000000; border-bottom: 1px solid #000000;padding-right: 10px; padding-left: 10px;}
</style>
'@

$sqlquery = @"
Use BES12
Select Top 1000000
  obj_user.display_name As [User],
  def_device_os_family.company_name as [Manufacturer],
  def_device_hardware.model as [Model],
  def_device_os.name as [OS Version],
  obj_device.normalized_phone_number as [Phone Number],
  Convert(Varchar(10), obj_user_device.last_communication, 102) As [Last Contact]
From obj_user
  Inner Join obj_user_device On obj_user_device.id_user = obj_user.id_user
  Inner Join obj_device On obj_device.id_device = obj_user_device.id_device
  Inner Join def_device_hardware on def_device_hardware.id_device_hardware = obj_device.id_device_hardware
  Inner Join def_device_os on def_device_os.id_device_os = obj_device.id_device_os
  Inner Join def_device_os_family on def_device_os_family.id_device_os_family = def_device_os.id_device_os_family
Order by [User]
"@

$message.Body = Invoke-Sqlcmd -Query $sqlquery -ServerInstance '.' |  Select * -ExcludeProperty RowError, RowState, Table, ItemArray, HasErrors | sort-object "Last Contact" | convertto-html -Head $style
$smtp = New-Object Net.Mail.SmtpClient($smtpServer)
$smtp.Send($message)

HOWTO: Scheduled a standalone VMware ESXi Host Reboot via Powershell

We have several clients who have standalone VMware ESXi hosts (that are not part of any vCenter) without any option for vMotion or Storage vMotion.  This can make it difficult for us to keep those hosts current with patches, updates, and BIOS / firmware because it means we need to manually shut the hosts’ guest down, and then restart the host – none of which can be done during normal business hours – and I’m getting too old to work all night.

Fortunately, VMware provides us a way to use PowerShell to shutdown a ESXi host’s guest, and then force a reboot.  This means we can apply patches and updates late in the day to the ESXi host, then schedule the host to reboot early in the morning after the daily backup completes.  Then when we come into the office in the morning (usually an hour or two before the clients arrive at their offices), it is simply a matter of checking the host to ensure it is back up along with all it’s guests.

To schedule a standalone VMware Host reboot, the current VMware PowerCLI client needs to be installed on the machine that will be running the scheduled reboot.

Once the VMware PowerCLI is installed, you need to create 3 files:

  • C:\WINDOWS\VMWARE_ROOT.PWD – encrypted file that contains the root user’s password
  • C:\WINDOWS\VMWARE_HOST_REBOOT.CMD – the wrapper that will call PowerShell from TaskScheduler
  • C:\WINDOWS\VMWARE_HOST_REBOOT.PS1 – the actual PowerShell script that executes the reboot

To create the file C:\WINDOWS\VMWARE_ROOT.PWD, open PowerShell and run the following command:

read-host -assecurestring "Enter Password" | convertfrom-securestring | out-file C:\WINDOWS\VMWARE_ROOT.PWD

 

At the “Enter Password” prompt, enter the password of the root user account for the ESXi host you want to reboot.

You also need to set the PowerShell Execution Policy to support remote signed scripts such as C:\WINDOWS\VMWARE_HOST_REBOOT.PS1.  To do this, in PowerShell run the following command and select Yes when prompted:

Set-ExecutionPolicy RemoteSigned

We need to schedule a time for VMWARE_HOST_REBOOT.CMD to run.  I’ve set 4:15 am local time on March 22, 2015 in the example shown below, but you can adjust as required.  In an administrative command prompt, run this (***note – this will create the scheduled task to run as the currently logged in user***):

schtasks /create /tn "VMware Host Reboot" /tr C:\WINDOWS\VMWARE_HOST_REBOOT.CMD /sc once /st 04:15:00 /sd 03/22/2015 /rp "*" /ru "%userdomain%\%username%"

Now we need to create C:\WINDOWS\VMWARE_HOST_REBOOT.CMD, which is the batch file task scheduler uses to launch our PowerShell script.

rem --- begin cut and paste of notepad C:\WINDOWS\VMWARE_HOST_REBOOT.CMD
@echo off
C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -ExecutionPolicy RemoteSigned -noprofile -File C:\WINDOWS\VMWARE_HOST_REBOOT.PS1
exit /b
rem --- end cut and paste of C:\WINDOWS\VMWARE_HOST_REBOOT.CMD ---

Lastly, we need to create C:\WINDOWS\VMWARE_HOST_REBOOT.PS1, adjusting the variable for $server to the host you wish to reboot (all variables are all defined at the top of the script) and adjust wait time ($waittime) before force rebooting after you issue a graceful gust shutdown command.

###--- begin cut and paste of notepad C:\WINDOWS\VMWARE_HOST_REBOOT.PS1
### VMWARE_HOST_REBOOT.PS1
### @deancolpitts – http://blog.jbgeek.net
### 2015.03.20

### This script will attempt to perform a graceful VM restart via the VMware Tools inside the guest.
### Variables - please only adjust server, user, and waittime. Any other variables should not be touched.
### Server is the vCenter server or ESXi host's FQDN, while user is the vCenter user or ESXi user account.

$server = "VMWARE.FQDN.DOMAIN_OR_IPADDRESS"
$user = "root"
$waittime = "300"

$credentialFile = "C:\WINDOWS\VMWARE_ROOT.PWD"
$pass = cat $credentialFile | convertto-securestring
$credentials = new-object -typename System.Management.Automation.PSCredential -argumentlist $user,$pass

add-pssnapin VMware.VimAutomation.Core -ErrorAction SilentlyContinue -WarningAction SilentlyContinue | Out-Null
if ( $DefaultVIServers.Length -lt 1 )
{
Connect-VIServer -Server $server -Protocol https -credential $credentials -WarningAction SilentlyContinue | Out-Null
}

Get-VM | Shutdown-VMGuest -confirm:$false -WarningAction SilentlyContinue

### Wait x number of seconds for all the VM's to gracefully shutdown before a forced kill occurs
Start-Sleep -s $waittime

Restart-VMHost -VMHost $server -force -confirm:$false

###--- end cut and paste of C:\WINDOWS\VMWARE_HOST_REBOOT.PS1 ---

All that is left do now is wait for C:\WINDOWS\VMWARE_HOST_REBOOT.CMD to run at your scheduled time.

As always – Use any tips, tricks, or scripts I post at your own risk.

KB3148812 breaks Windows Server Update Services

Earlier this week, Microsoft pushed out KB3148812, which enables ESD decryption provisioning in WSUS (on Windows Server 2012 and Windows Server 2012 R2).  From what I read and understand, KB3148812 is going to be a mandatory update for WSUS to support Windows 10 updates after May 1.  Unfortunately, it appears that KB3148812 also breaks the WSUS console.  Rather than re-issue KB3148812 so it doesn’t break WSUS, Microsoft has published in a blog posting the necessary post-install steps to “un-break” WSUS after install KB3148812.

Basically you need to perform two steps to return WSUS to a working condition on Windows 2012.  First, you need to re-run the WSUS post-install.  And then you also need to add HTTP Activation to your WSUS server.

For Step 1, to re-run post-install, open an administrative command prompt and run:

"C:\Program Files\Update Services\Tools\Wsusutil.exe" postinstall /servicing

For Step 2, to install HTTP Activation, open an administrative PowerShell command prompt and run:

Install-WindowsFeature -Name NET-WCF-HTTP-Activation45

No reboot should be necessary and the WSUS console should now open and function normally for you.  There is the possibility you’ll still get client connectivity issue though with an error of 0x80244007, which is something entirely different that Microsoft is still looking at as of 2016.04.22.

Setup hourly HPE Insight Remote Support Service checking

In a previous post, I mentioned we utilize HPE Insight Remote Support (IRS) at all our client sites, and discovered the lovely undocumented “feature” that IRS has, which is a tendency not to start after a Windows server reboot after an IRS update. This great undocumented feature defeats the entire purpose of IRS – monitoring and alerting your HPE hardware. After getting burned by this feature three or four times in a month where customers noticed hardware faults (via amber alert lights on the equipment) before we did since IRS was not running to alert us, I decided it was time to write a script to check IRS hourly and alert us if it wasn’t running.

To configure Windows to send an alert if the HP IRS Service is stopped, create the following two files (file contents are at the end of this post) on the IRS server:

  • check_irs_service_status.cmd – which is the wrapper that will call PowerShell from Task Scheduler
  • check_irs_service_status.ps1 – which is the actual PowerShell script that executes the service status check

Lastly, we need to schedule check_irs_service_status.cmd to run hourly. I’ve set 2 minutes after the hour in the example shown below, but you can adjust as required.

schtasks /create /tn "Hourly IRS Service Check" /tr c:\Windows\check_irs_service_status.cmd /sc minute /mo 60 /st 00:02:00 /rp "*" /ru "%userdomain%\%username%"

By default, the SMTP from address will be the netbios computer name of the IRS server @ the User’s DNS Domain FQDN (i.e. IRS-SERVER@JBGEEK.NET).  The SMTP to address will be support @ the User’s DNS Domain FQDN (i.e. SUPPORT@JBGEEK.NET), and the SMTP server will be mail @ the User’s DNS Domain FQDN (i.e. MAIL.JBGEEK.NET).  You can determine what these will be by checking the system’s environment variables with SET from a command prompt.  You can customize these settings in the “Send-MailMessage” command if necessary.

All that is left to do is to stop the service and test run check_irs_service_status.cmd to verify the Send-MailMessage works properly in your environment.

 

check_irs_service_status.cmd

rem --- begin cut and paste of notepad c:\windows\check_irs_service_status.cmd
@echo off
C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -ExecutionPolicy RemoteSigned -noprofile -File C:\Windows\check_irs_service_status.ps1
exit /b
rem --- end cut and paste of c:\windows\check_irs_service_status.cmd ---

 

check_irs_service_status.ps1

###--- begin cut and paste of notepad c:\windows\check_irs_service_status.ps1
### Check_irs_service_status.ps1
### @deancolpitts – http://blog.jbgeek.net
### 2016.01.27
### This script will check the status of the server HPRSMAIN and alert via email if the service is stopped.

$Service = Get-Service -name HPRSMAIN
$Service.Status
if ($Service.Status -eq "Stopped") {
 $CurrentTime = Get-Date
 Send-MailMessage -From "$env:computername@$env:userdnsdomain" -To "support@$env:userdnsdomain" -Subject "$env:computername - HP IRS Service is stopped!!!" -Body "The HP IRS Service is stopped on $env:computername.$env:userdnsdomain at approximately $CurrentTime." -Priority High -DNO onSuccess, onFailure -SmtpServer "mail.$env:userdnsdomain"
}

###--- end cut and paste of notepad c:\windows\check_irs_service_status.ps1

 

HOWTO: Schedule Daily Netscaler VPX Reboots via Powershell

We often utilize Citrix’s NetScaler VPX running on VMware ESXi 5.5 to allow our clients to securely connect to their Citrix infrastructure from outside the firewall.  For the most part – it works well.  Unfortunately though, our experience has taught us that occasionally NSVPX goes all fubar on it’s own after a few days of running and stops processing connection requests once the user logs in.  A simple reboot of the NSVPX VM usually resolves the user’s connectivity issues..

To combat this issue, I wrote a Powershell script that we run as a daily scheduled task on our management server to have vCenter automatically restart the machine once a day.  You could easily modify this script to reboot any VM you want though.

To configure daily VM rebooting, the current VMware PowerCLI client needs to be installed on the machine that will be running the scheduled reboot.  Once the VMware PowerCLI is installed, you need to create 3 files on the management machine:

  1. daily_nsvpx_reboot.cmd – which is the wrapper that will call PowerShell from TaskScheduler (see below in for cut and paste of the file contents)
  2. daily_nsvpx_reboot.ps1 – which is the actual PowerShell script that executes the reboot (see below for cut and paste of the file contents)
  3. daily_nsvpx_reboot.pwd – which is an encrypted file that contains the vCenter user’s password

To create the file daily_nsvpx_reboot.pwd, open PowerShell and run the following command:

read-host -assecurestring "Enter Password" | convertfrom-securestring | out-file c:\windows\daily_nsvpx_reboot.pwd

At the “Enter Password” prompt, enter the password of the user account you will be using that has rights in vCenter or the ESXi host to perform VM restarts.

You may also need to set the PowerShell Execution Policy to support remote signed scripts such as daily_nsvpx_reboot.ps1.  To do this, open PowerShell and run the following command and select Yes when prompted:

Set-ExecutionPolicy RemoteSigned

After creating daily_nsvpx_reboot.cmd and daily_nsvpx_reboot.ps1 (see below for file contents of these two files), edit daily_nsvpx_reboot.ps1 and adjust the variables for $server, $user, and $vm2reboot to fit your environment (these three variables are all defined at the top of the script).

Lastly, you need to schedule daily_nsvpx_reboot.cmd to run daily.  I’ve set 4:15 am local time in the example shown below, but you can adjust as required.  To schedule the task, open an Administrative command prompt and run the following command (adjust domain\username to be the same user account that has rights in vCenter or the ESXi host to perform VM restarts):

schtasks /create /tn "Daily NSVPX Reboot" /tr C:\WINDOWS\DAILY_NSVPX_REBOOT.CMD /sc daily /st 04:15:00 /rp "*" /ru "domain\username"

All that is left do now is test run daily_nsvpx_reboot.cmd and see that it runs and reboots the NSVPX.  If you are monitoring via ProcExp or TaskManager on the management machine, you should note low CPU usage followed by several spikes up to 50% (it is single threaded), and you should be able to see in the NSVPX console via vCenter when it reboots.

And as always – Use any tips, tricks, or scripts I post at your own risk.


daily_nsvpx_reboot.cmd – file contents

rem — begin cut and paste of notepad c:\windows\daily_nsvpx_reboot.cmd
@echo off
C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -ExecutionPolicy RemoteSigned -noprofile -File C:\Windows\daily_nsvpx_reboot.ps1
exit /b
rem — end cut and paste of c:\windows\daily_nsvpx_reboot.cmd —

daily_nsvpx_reboot.ps1 – file contents

    ###— begin cut and paste of notepad c:\windows\daily_nsvpx_reboot.ps1

    ### Daily_nsvpx_reboot.ps1
    ### @deancolpitts – http://blog.jbgeek.net
    ### 2015.01.02
    ### This script will attempt to perform a graceful VM restart via the VMware Tools inside the guest.

    ### Variables – please only adjust server, user, and vm2reboot.  Any other variables should not be touched.
    ### Server is the vCenter server or ESXi host’s FQDN, while user is the vCenter user or ESXi user account.
    ### if any smtp variables present, they should be self-explanatory.

    $server = “vcenter.domain.fqdn”
    $user = “vcenter_username”
    $vm2reboot = “nsvpx”

    ### Read the encrypted user password from “c:\windows\daily_nsvpx_reboot.pwd”
    ### Use the following commented out PowerShell command to manually create a new credentials store.
    ### Enter the user’s password when prompted while running the read-host command
    ### read-host -assecurestring “Enter Password” | convertfrom-securestring | out-file c:\windows\daily_nsvpx_reboot.pwd

    $credentialFile = “c:\windows\daily_nsvpx_reboot.pwd”
    $pass = cat $credentialFile | convertto-securestring
    $credentials = new-object -typename System.Management.Automation.PSCredential -argumentlist $user,$pass

    add-pssnapin VMware.VimAutomation.Core -ErrorAction SilentlyContinue -WarningAction SilentlyContinue | Out-Null

    if ( $DefaultVIServers.Length -lt 1 )
    {
    Connect-VIServer -Server $server -Protocol https -credential $credentials -WarningAction SilentlyContinue | Out-Null
    }

    Restart-VM -VM $vm2reboot -RunAsync -Confirm:$false

    ###— end cut and paste of c:\windows\daily_nsvpx_reboot.ps1 —