HOWTO: HTTP boot the HPE Proliant Service Pack ISO DVD using RESTfulAPI to update firmware without messing with WDS or PXE

Most of my customer sites consist of one to four HPE Proliant DL3xx servers running VMware ESXi and an additional HPE Proliant DL3xx running Windows 2012 R2 / 2016. HPE offers some great tools for managing their servers, but unfortunately for smaller organizations, most of HPE’s management tools (and I’m looking squarely at you Insight Control and OneView) take more time to setup and get running correctly then the time you’ll save by installing / updating a small handful of servers manually.  Therefore, I usually don’t deploy these tools to help install OSes or update firmware at my smaller client sites.  I generally just rely on booting the HPE Support Pack for Proliant (SPP) to update firmware, use a USB key with a scripted ESXi install on it for installing ESXi, and utilize WDS to install Windows directly on my Proliants when required.

Prior to HPE Proliant Gen 9 servers, I would PXE boot the Proliant Service Pack using PXELINUX and mount the ISO via NFS.  Then along came Gen 9 with UEFI.  Unfortunately, PXELINUX suffers from a complete lack of support for UEFI.  A couple of times I pestered some of the HPE SPP developers and managers in person while at HPE’s campus in Houston, but they never really showed much interest in explaining or documenting how to get network booting working with the SPP when the server utilized UEFI, so I had pretty much given up on ever getting it to work.

The other day I was playing with the HPE RESTful Interface Tool and decided to try configuring HTTP boot on DL380 Gen10 with the current SPP ISO image (P11740_001_spp-2018.11.0-SPP2018110.2018_1114.38.iso).  Much to my surprise, after modifying only a single configuration file on the ISO image, I was able to successfully boot the current SPP ISO image via HTTP and run a full firmware update on the Gen10 I was playing with.

The nice thing about this method is that because it is all done via HTTP, you don’t have to mess with or disable your WDS (Windows Deployment Services) server to add Linux support (which is what the SPP ISO is based on).  So this is great news for pure Windows shops!  And as a bonus, these steps works with Gen 9 servers too.

So how did I do it?  Before I share that, as always:

Use any tips, tricks, or scripts I post at your own risk.

First, you need to slightly modify the SPP ISO image.  Copy the original SPP ISO image to your web server (i.e. c:\inetpub\wwwroot).

Open the ISO image with your favorite ISO editor and extract \efi\boot\grub.cfg, then open the grub.cfg with a decent text editor (i.e. Notepad++, but definitely not the built-in Windows Notepad).  Scroll down the first menuentry, which will be “Automatic Firmware Update”.  Then copy and paste the following just above that menuentry:

menuentry "HTTP Firmware Update Version 2018.11.0" {
set gfxpayload=keep
echo "Loading kernel..."
linux /pxe/spp2018110/vmlinuz media=net root=/dev/ram0 ramdisk_size=10485760 init=/bin/init  iso1=http://xxx.xxx.xxx.xxx/spp.iso iso1mnt=/mnt/bootdevice hp_fibre cdcache TYPE=MANUAL AUTOPOWEROFFONSUCCESS=no modprobe.blacklist=aacraid,mpt3sas  ${linuxconsole}
echo "Loading initial ramdisk..."
initrd /pxe/spp2018110/initrd.img
}

So your grub.cfg will look like this when you are done:

2018.12.20 - 17.45.17 - SNAGIT - 0027

Adjust the http address (xxx.xxx.xxx.xxx), path, and ISO image name as required for your network, then save the updated grub.cfg and inject it back into the ISO image, over-writing the existing \efi\boot\grub.cfg, and then save the updated ISO image.

Be sure to add the .ISO mime type to your web server so that the ISO file type can be handled correctly.  The command below will work with IIS 8.5 and above to add a new mime type to IIS for .ISO.

C:\Windows\System32\inetsrv\appcmd.exe set config -section:system.webServer/staticContent /+"[fileExtension='iso',mimeType='application/iso']"

Now, you need to install the HPE RESTful Interface Tool on your machine.  The current version at the time of this writing is 2.3.4.0.  Go to the Hewlett Packard Enterprise Support Center and search for “RESTful Interface Tool for Windows”, then download and install the .msi (there is a Linux version available as well there).

Once the HPE RESTful Interface Tool is installed, run it as an Administrator.  Next, you need to connect to your server’s ILO, select the Bios object, set the UrlBootfile Entry and commit the changes.

*** NOTE: Make sure the UrlBootFile entry matches the url of your ISO image that your put on your webserver and specified as the iso1 switch in the grub.cfg entry.

ilorest
login ilo_ip_address -u admin -p password
select Bios.v1_0.0
set UrlBootFile=http://xxx.xxx.xxx.xxx/spp.iso
commit

2018.12.19 - 13.56.41 - SNAGIT - 0003

This takes care of the changes you must make to your Proliant server (keep in mind each server that you want to HTTP boot needs to have this this done).

The next time your server boots, the UrlBootFile change will be applied at the end of POST, then server will automatically reboot and start to POST again.

2018.12.19 - 14.18.08 - SNAGIT - 0005

That’s it – your configuration is all done.  Now when you reboot your server, if you hit F11 for the Boot Menu, you’ll have an entry for HTTP there – select it.

2018.12.19 - 14.20.01 - SNAGIT - 0006

After maybe 30 to 45 seconds (depending on your network speed – I’m using 10GbE), you’ll see the familiar SPP boot menu, but with an extra entry which is set as the default entry.

2018.12.19 - 14.21.25 - SNAGIT - 0009

Select it, and after about a minute (again – I’m using 10GbE) you’ll see the ISO image get mounted.

2018-12-20_17-54-25

If the image fails to mount, verify you are able to download the image you specified as the UrlBootFile from your PC.  If that works, then verify that the grub.cfg is correctly updated, with no typos.  Also – verify your server has 16GB+ of RAM in it, as the grub entry creates a 10GB RAM disk.  You may also need to upgrade the ILO firmware and drivers to current builds (such as 2.61 for ILO4 or 1.39 for ILO5) before using the iLOrest tool.

If you so desire, you could also set the new grub entry to be totally automatic by grabbing the proper switches out of the “Automatic Firmware Update” entry.  I suspect it may also be possible to split the ISO and boot one ISO without the packages folder (so it boots quicker) and mount a second the ISO with the packages folders still there to run the upgrades from.  Just to be clear, I haven’t tested that yet – it’s just a theory at this point.

I have tested this by HTTP booting over a branch office VPN tunnel which tops out at 100Mbps – it took a while for the image to load (I didn’t time it as I was working on other things at the time), but it did eventually load and it successfully updated the remote server.

When the next Support Pack for Proliant is released, all you need to do is update the grub.cfg with the correct paths and copy the updated ISO to your webserver with the same file name you used here.  You shouldn’t need to adjust the UrlBootFile on your servers.

Happy updating!

 

 

HOWTO: Set the creation and modification timestamp on a file via #PowerShell

Recently, I updated one of our internal tool kits, and then packaged it for distribution.  It was a busy day when I updated it, so I didn’t manage to package it on the same day as I had updated / built / compiled it.  Internally, we use the date as the version number of the tool (occasionally suffixed with a letter which indicates my screw-ups in the build process on that given day).  In this particular case, the version number was 2018-11-24b, indicating I updated it on 2018-11-24, and that this was the 3 revision (no suffix, a, then b) that I had created on 2018-11-24 (I found bugs in the first two after testing the packaging).

Because I wasn’t packaging on the same day as I updated it, the time stamps on my archives didn’t match the build date, so I need to change them – all of them!  So I figured up PowerShell and used it instead.  Below are the commands necessary to view and set both the creation and modification timestamps on a file via an elevated PowerShell prompt.

As always – Use any tips, tricks, or scripts I post at your own risk.

To view the file creation timestamp:

(Get-ChildItem “c:\path\file_to_change.wim”).CreationTime

To set the file creation timestamp:

(Get-ChildItem “c:\path\file_to_change.wim”).CreationTime = ’11/24/2018 11:24AM’

To view the file modification timestamp:

(Get-ChildItem “c:\path\file_to_change.wim”).LastWriteTime

To set the file modification timestamp:

(Get-ChildItem “c:\path\file_to_change.wim”).LastWriteTime = ’11/24/2018 11:24AM’

To set the creation and modification timestamp on every single file in a folder:

foreach ($objFile in Get-ChildItem “c:\path\*”) {$objFile.Creationtime = ’11/24/2018 11:24AM’}

foreach ($objFile in Get-ChildItem “c:\path\*”) {$objFile.LastWriteTime = ’11/24/2018 11:24AM’}

 

 

HOWTO: Permanently replace the ugly Windows 10/2016 login screen background and colors for all users with #PowerShell

I can’t stand the default Windows 10 and Windows Server 2016 logon background, and one of the first things I do when I build a new Windows template at a customer site is wipe that default background out!  I typically replace it with a single solid color, and I’m kind of fond of the old blue backgrounds that came with Windows XP (or was it Windows NT 4 – or may Windows 2000, I don’t remember now) as they are easy on the eyes… Anyways – the background color I like and use has a RGB value of 58 110 165.

I used to have a basic batch file to wipe it Microsoft’s stock background out by copying an existing background over from my staging server, but with every iteration of Windows 10 and Windows Server 2016, the path to img100.jpg in C:\Windows\WinSxS changes.  So last night I decided it was time to use some PowerShell to take care of this menace and allow the script to run on multiple platforms and software updates.

I struggled with creating a new solid color background jpg in PowerShell using the RGB value I wanted, but eventually I found some code that someone had posted elsewhere on how to create a gradient jpg, so I snagged it and set the gradient to be same at the end as the beginning, which results in a solid color all the way across.  I’m sure someone with better skills than me could clean this up properly – but this suits my purposes for what I need so I stopped searching for a better way.

So basically what this script does is create a new jpg that is 640×480 in C:\Windows\Web\Wallpaper\Staging, adjusts the accent colors for the current user and the default user profile, finds the path to img100.jpg and replaces it after taking ownership and setting appropriate ntfs rights to it, then clears out the lock screen jpgs using RoboCopy.  The lock screen jpgs are owned by the System account, and Robocopy /mir /zb is the simplest way to wipe them out that I know of without using Sysinternals Suite psexec to involve System account privileges and delete the jpgs.

You definitely need to run this in an elevated PowerShell session too!

As always – Use any tips, tricks, or scripts I post at your own risk.

New-Item -Path "C:\Windows\Web\Wallpaper\Staging" -ItemType "Directory" -Force -Confirm:$false | out-null
Add-Type -AssemblyName System.Drawing
$newbackground = New-Object System.Drawing.Bitmap 640, 480
[System.Drawing.Graphics]::FromImage($newbackground).FillRectangle(
(New-Object System.Drawing.Drawing2D.LinearGradientBrush(
(New-Object System.Drawing.Point(0, 0)),
(New-Object System.Drawing.Point(640, 480)),
[System.Drawing.Color]::FromArgb(58, 110, 165),
[System.Drawing.Color]::FromArgb(58, 110, 165))),
0, 0, $newbackground.Width, $newbackground.Height)
$newbackground.Save('C:\Windows\Web\Wallpaper\Staging\background.jpg',[System.Drawing.Imaging.ImageFormat]::Jpeg)
copy-item -path C:\Windows\Web\Wallpaper\Staging\background.jpg -destination c:\windows\web\wallpaper\background.jpg -force -confirm:$false
REG LOAD HKEY_USERS\ZZZ C:\USERS\DEFAULT\NTUSER.DAT
REG ADD "HKEY_USERS\ZZZ\SOFTWARE\Microsoft\Windows\CurrentVersion\Explorer\Accent" /f /v "StartColor" /t REG_DWORD /d 0xffa66c39
REG ADD "HKEY_USERS\ZZZ\SOFTWARE\Microsoft\Windows\CurrentVersion\Explorer\Accent" /f /v "AccentColor" /t REG_DWORD /d 0xffb51746
REG UNLOAD HKEY_USERS\ZZZ
REG ADD "HKEY_USERS\.DEFAULT\SOFTWARE\Microsoft\Windows\CurrentVersion\Explorer\Accent" /f /v "StartColor" /t REG_DWORD /d 0xffa66c39
REG ADD "HKEY_USERS\.DEFAULT\SOFTWARE\Microsoft\Windows\CurrentVersion\Explorer\Accent" /f /v "AccentColor" /t REG_DWORD /d 0xffb51746
REG ADD "HKEY_CURRENT_USER\SOFTWARE\Microsoft\Windows\CurrentVersion\Explorer\Accent" /f /v "StartColor" /t REG_DWORD /d 0xffa66c39
REG ADD "HKEY_CURRENT_USER\SOFTWARE\Microsoft\Windows\CurrentVersion\Explorer\Accent" /f /v "AccentColor" /t REG_DWORD /d 0xffb51746
REG ADD "HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Explorer\Accent" /f /v "DefaultStartColor" /t REG_DWORD /d 0xffa66c39
takeown /f C:\ProgramData\Microsoft\Windows\SystemData /a /r /d y
takeown /f C:\Windows\Web\Screen\img100.jpg /a
icacls C:\Windows\Web\Screen\img100.jpg /grant Administrators:F
$lockscreen = "C:\ProgramData\Microsoft\Windows\SystemData\S-1-5-18\ReadOnly\LockScreen_Z"
$tempfolder = "C:\ProgramData\Microsoft\Windows\SystemData\S-1-5-18\ReadOnly\LockScreen_Temp"
$img100 = Get-ChildItem C:\Windows\WinSxS -Recurse -Include img100.jpg
write-host $img100
takeown /f $img100 /a
icacls $img100 /grant Administrators:F /q
copy-item -path c:\windows\web\wallpaper\background.jpg -destination $img100 -force -confirm:$false | out-null
copy-item -path c:\windows\web\wallpaper\background.jpg -destination C:\Windows\Web\Wallpaper\Windows\BlueBackground.jpg -force -confirm:$false | out-null
copy-item -path c:\windows\web\wallpaper\background.jpg -destination C:\Windows\Web\Screen\img100.jpg -force -confirm:$false | out-null
New-Item -Path $tempfolder -ItemType "Directory" | out-null
Robocopy $tempfolder $lockscreen /zb /mir /njh /njs
Remove-Item -Path $tempfolder -force -confirm:$false | out-null

 

HOWTO: #PowerShell script to download, extract and add #SysinternalsSuite to the path

I absolutely love Microsoft’s Sysinternals Suite – it’s an amazing set of tools for troubleshooting and tweaking Windows machines.  Heck – there isn’t a day goes by that I don’t use at least one of the tools out of the suite.  I generally try to download, extract and add the suite to the path of any computer I touch.

This morning while building a new 2016 template for a customer, I realized I had missed downloading and adding it to the path, but the VM was in a firewalled VLAN and unable to reach my staging and support server – so I couldn’t just grab the extracted directory from my staging server.  This got me to thinking there must be a simple way to use a cli or script to download, extract, and add the extracted folder to the computer’s path.  So I took 30 minutes and wrote one.

Basically, this script can be cut and pasted into an elevated PowerShell session, and it will grab the most recent SysinternalsSuite.zip from Microsoft, extract the .zip to C:\Program Files\SysinternalsSuite, and then add C:\Program Files\SysinternalsSuite to the computer’s path if it does not already exist in the path.

I’ve tested this with Windows 7, Windows 10 (1803), Windows Server 2012 R2 and Windows Server 2016.

As always – Use any tips, tricks, or scripts I post at your own risk.

Import-Module BitsTransfer
$url_zip = "https://download.sysinternals.com/files/SysinternalsSuite.zip"
$output_path = "C:\Program Files\SysinternalsSuite"
$output_zip = $output_path + '\SysinternalsSuite.zip'
Remove-Item -Path $output_path\*.* -force -confirm:$false
New-Item -Path $output_path -ItemType "Directory" -Force -Confirm:$false | out-null
Start-BitsTransfer -Source $url_zip -Destination $output_zip
Add-Type -AssemblyName System.IO.Compression.FileSystem
function Unzip
{
param([string]$zipfile, [string]$outpath)
[System.IO.Compression.ZipFile]::ExtractToDirectory($zipfile, $outpath)
}
Unzip $output_zip $output_path
Remove-Item -Path $output_zip -force -confirm:$false
$oldpath = (Get-ItemProperty -Path "Registry::HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Session Manager\Environment" -Name PATH).path
If ($oldpath -NotLike "*SysinternalsSuite*") {
$newpath = "$oldpath;$output_path"
Set-ItemProperty -Path "Registry::HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Session Manager\Environment" -Name PATH -Value $newPath
}
$writtenpath = (Get-ItemProperty -Path "Registry::HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Session Manager\Environment" -Name PATH).path
write-host $writtenpath

 

 

HOWTO: Send “Shift + F10” to a remote #VMware / #vPro console over #RDP

We typically do a lot of our work at client locations over RDP to a server that resides on-prem in the client’s data center.  We then use that on-prem server as a jump server to manage other systems that reside on-prem and for the most part it works well except for the occasion keyboard combo press that just won’t go through to the client machine we are remoted into via the RDP jump server.  This is particularly a problem is when we are deploying Windows images to machines, which we do either via Intel’s vPro KVM controls (if it is physical) or via VMware Console (if it is virtual).  Occasionally the Windows machine will fail to boot during the specialize phase of sysprep, and we need to troubleshoot the issue.

If we were physically sitting in front of the machine, the process would be pretty simple – hit Shift + F10 and a command prompt pops open, and from there you navigate to C:\Windows\Panther and access the sysprep logs using Notepad.  But in our case, because we are utilizing a jump server via RDP to access the console (either vPro or VMware), Shift + F10 is being intercept by the jump server and not passed on to the vPro KVM session or the VMware Console, which means we can’t get to a command prompt to start troubleshooting.  When this happens, we need to disconnect from the RDP session, and use either RealVNC Plus (for the vPro console) or the VMware client directly from our local machines over VPN, which in some cases is deathly slow at best.

After getting stuck the other day having to troubleshoot a sysprep error over VPN using vPro instead of RDP using vPro, I decided there had to be a way to script a hotkey to send Shift + F10 to the console via RDP.  Unfortunately, I didn’t find anything readily available, so I scripted my own using AutoIt, which I then compiled into an .exe and digitally signed with my code signing certificate.

Basically the script searches for a window that has a title of “Intel(r) AMT KVM – VNC Viewer Plus” or a window that has a title that contains ” on ” (as in Windows7 on ESXiHost” and makes the first instance it finds the active window, then sends a Shift + F10 to the window.

Now when I have to send a Shift + F10 to a remote console during troubleshooting, I simply run the correct executable on our RDP jump servers and up comes the command prompt in the remote console!

Below are the two AutoIt scripts and further down are the two compiled .exe files.

As always – Use any tips, tricks, or scripts I post at your own risk.

Code for Intel(r) AMT KVM – VNC Viewer Plus:

$Title = "Intel(r) AMT KVM - VNC Viewer Plus"
WinWait ($Title)
WinActivate ($Title)
Send ("+{F10}")
Sleep (100)
Exit

Code for VMware Console:

Opt("MouseCoordMode", 0)
Opt("WinTitleMatchMode", 2)
$Title = "[REGEXPTITLE:(?i)(.* on .*)]"

WinWait ($Title)
WinActivate ($Title)

MouseClick ( "right",9,88,1 )
Send ("+{F10}")
Sleep (100)
Exit

Already compiled and digitally signed AutoIT executables:

SendShiftF10toVMware
SendShiftF10tovPro

#Blackberry UEM Server – Exporting the serial numbers of all devices

We use Blackberry’s UEM 12.7.2 to manage all our mobile devices and all our clients’ mobile devices – be it Blackberry, Apple, Android, or even (shudder…) Windows Phone. Recently a client asked me for a list of all his devices with the serial numbers. Since there were only 30 devices or so, I figured I’d just go into the device report for each user and cut and paste the serial number into a spreadsheet and send it to the customer. Wow – was I ever wrong. The serial number isn’t reported in the Device Report, nor is on the user’s device tab. The only place I could actually find it listed is on the All Users tab if you enable Advanced View, but that is useless because cut and paste is disabled on this page.

But since the serial number appears there, then it must be in the database.

So, I went back to one of my previous blog posts about how to script and report the last contact time of a device (see that blog post here) and did a bit of modification to it.  And since I didn’t know which table the serial number was listed in, I used some code that I found elsewhere (I don’t remember where, it was a while ago, so I’m not able to link to it, and I won’t republish it since I don’t know who to give credit to) to search all the database tables for one of the serial number that I manually copied down.

Turns out the device serial number is stored in the table obj_device_setting, and that id_device_setting_definition “60” is the device’s serial number.  So knowing that, I was able to create a new query based on my previous blog post that now also includes the device serial number.

Here is the updated SQL query to include device serial numbers.

Select Top 1000000
obj_user.display_name As [User],
def_device_os_family.company_name as [Manufacturer],
def_device_hardware.model as [Model],
def_device_os.name as [OS Version],
obj_device.normalized_phone_number as [Phone Number],
obj_device_setting.value as [Device Serial Number],
Convert(Varchar(10), obj_user_device.last_communication, 102) As [Last Contact]
From obj_user
Inner Join obj_user_device On obj_user_device.id_user = obj_user.id_user
Inner Join obj_device On obj_device.id_device = obj_user_device.id_device
Inner Join def_device_hardware on def_device_hardware.id_device_hardware = obj_device.id_device_hardware
Inner Join def_device_os on def_device_os.id_device_os = obj_device.id_device_os
Inner Join def_device_os_family on def_device_os_family.id_device_os_family = def_device_os.id_device_os_family
Inner Join obj_device_setting On obj_device_setting.id_device = obj_user_device.id_device
Where obj_device_setting.id_device_setting_definition = '60'
Order by [User]

As always – Use any tips, tricks, or scripts I post at your own risk.

HOWTO: Using #PowerShell to ensure a #Veeam USB Repository always has the correct drive letter

Some time ago I had a customer who switched completely to Veeam from Backup Exec (yeah baby!!!).  Offsite replication of any sort was out of the question as the customer simply couldn’t get the necessary bandwidth from any of the ISPs that serviced the area at anything that even approached an unreasonable price.  Instead we opted for a HPE StoreOnce for onsite and some sort of removable drive system for daily offsite.  The customer insisted on using Western Digital My Books (USB3) due to their capacity (6TB) and semi-ruggedness for the offline repo.  The drives are rotated out each day by a staff member for a new drive and they send the previous night’s drive offsite.

The My Books were originally setup using F: as the drive letter.  This would work great until someone plugged a USB key into the server and it was auto-assigned F: by Windows 2012R2, or if someone logged in who had a drive mapping of F:.  And then other days, Windows would randomly assign a drive letter other than F: even though F: was available.  At this point, Veeam would puke because it couldn’t find the repo anymore.  And occasionally the customer would forget to swap the drive for a fresh one (or even correct drive some days), which meant there wouldn’t be enough free space on the repo to complete that night’s backup (about 4.9TB is processed daily).

To work around all this, I ended up writing a PowerShell script to always map the My Book to Z:, clean the previous backups off of it (if any were found, or if the wrong drive with a different backup set was plugged in), and start the actual backup job.  And if the script can’t find the My Book or free disk capacity is less than 95% after attempting cleanup, it will abort the backup and send an email warning to the support team that the backup was aborted.  Rather than have Veeam schedule the job to run, I use Windows Task Scheduler instead to run the script, then use the Veeam PowerShell module to start the backup job from inside the script after I’ve verified Z: is present.

This script assumes that every rotated My Book drive has been formatted and has a volume label containing “WD MY BOOK”.  It also assumes the SMTP server is called mail dot whatever the DNS domain name of the machine running the script is (i.e. my AD is jbgeek.net, so mail.jbgeek.net).  All of my client sites have a cname for their Exchange server called “mail” created in their AD DNS zone – which means I can cut and paste the same script without modification from one client site to another, which cuts down the chance of an editing error, and makes it quicker to deploy new scripts when necessary for my team.

You will need to adjust the “Get-ChildItem -Path” (lines 21 to 26) directories to fit your environment, along with the $SendEmailTo variable (line 7) and the “Start-VBRJob -Job” job name (line 41).  Line 28 defines the cutoff point for disk capacity – if the available disk space is less than 95% of the drive capacity, the backup aborts.  Line 1 is commented it out – I generally always start my scripts with # start notepad++ script.filename so that I can just cut and paste it into a command prompt to create my script file – that way as I on-board each new customer, their sites are configured the same as existing sites.  Again, it cuts down the chances of an error and makes it quicker for my guys to deploy new scripts.

**NOTE – the following deletes data from your backup cartridges. Use any tips, tricks, or scripts I post at your own risk.  I accept zero liability and responsibility if you use these scripts!!!**

Here is the command line to create the scheduled task to run as the logged in user at 10pm each weekday (it will prompt you for your password when you run it in a command prompt):

schtasks /create /tn "Weekday Veeam USB Drive Backup" /tr "\"C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe\" -ExecutionPolicy RemoteSigned -noprofile -File  C:\Windows\Run_External_Drive_Veeam_Backup.ps1" /sc daily /st 22:00:00  /rp "*" /ru "%userdomain%\%username%"

Here is the Powershell script (updated 2017.01.17 to fix an error):

# start notepad++ C:\Windows\Run_External_Drive_Veeam_Backup.ps1

add-pssnapin Veeam*

$DnsDomain = Get-WmiObject -Class Win32_NTDomain -Filter "DSDirectoryServiceFlag='True'" | Select -ExpandProperty DnsForestName
$ThisComputerName = Get-WmiObject -Class Win32_ComputerSystem | Select -ExpandProperty Name
$SupportEmailAddress = "support@$DnsDomain"

# Get-WmiObject Win32_Volume | Where-Object {$_.DriveLetter -eq $null}
$drives = Get-WmiObject -Class win32_volume | Where-Object {$_.Label -match "WD MY BOOK"}
$i = 0
Foreach($drive in $drives)
{
#Set letter
$DriveLetter = "Z:"
Set-WmiInstance -input $drive -Arguments @{DriveLetter="$DriveLetter"}
$i++
}

if (Test-Path -path Z:\){
Get-ChildItem -Path "Z:\Veeam\Backups\All VMs - Z Drive" -Include *.vbm -recurse | foreach { $_.Delete()}
Get-ChildItem -Path "Z:\Veeam\Backups\All VMs - Z Drive" -Include *.vib -recurse | foreach { $_.Delete()}
Get-ChildItem -Path "Z:\Veeam\Backups\All VMs - Z Drive" -Include *.vbk -recurse | foreach { $_.Delete()}
Get-ChildItem -Path "Z:\Veeam\Backups\All VMs" -Include *.vbm -recurse | foreach { $_.Delete()}
Get-ChildItem -Path "Z:\Veeam\Backups\All VMs" -Include *.vib -recurse | foreach { $_.Delete()}
Get-ChildItem -Path "Z:\Veeam\Backups\All VMs" -Include *.vbk -recurse | foreach { $_.Delete()}
$externaldrivelabel = Get-WmiObject -Class win32_volume | Where-Object {$_.Label -match "WD MY BOOK"} | Select -ExpandProperty label
Get-WmiObject -Class win32_volume | Where-Object {$_.Label -match "WD MY BOOK"} | % { if (($_.FreeSpace/$_.Capacity) -le '0.95' )
{Send-MailMessage -From "$($ThisComputerName.ToUpper())@$($DnsDomain.ToUpper())" -To "$SupportEmailAddress" `
-Subject "Backup Aborted due to low space on $($externaldrivelabel)." `
-Body  "Aborting nightly Veeam backup to $($externaldrivelabel) (Z:) on $($ThisComputerName.ToUpper()).$($DnsDomain.ToUpper()) due to low disk space." `
-Priority High -DNO onSuccess, onFailure -SmtpServer "mail.$DnsDomain"
break}
}
}
if (Test-Path -path Z:\){
Send-MailMessage -From "$($ThisComputerName.ToUpper())@$($DnsDomain.ToUpper())" -To "$SupportEmailAddress" `
-Subject "Now starting nightly Veeam backup to $($externaldrivelabel)." `
-Body  "Now starting nightly Veeam backup to $($externaldrivelabel) (Z:) on $($ThisComputerName.ToUpper()).$($DnsDomain.ToUpper())." `
-Priority High -DNO onSuccess, onFailure -SmtpServer "mail.$DnsDomain"
Start-VBRJob -Job "All VMs - Z Drive" -FullBackup -RunAsync
break
}else{
Send-MailMessage -From "$($ThisComputerName.ToUpper())@$($DnsDomain.ToUpper())" -To "$SupportEmailAddress" `
-Subject "Error:  No external backup drive (Z:) found on $($ThisComputerName.ToUpper()).$($DnsDomain.ToUpper())!!!" `
-Body  "Error:  Cannot find or mount the external backup drive (Z:) on $($ThisComputerName.ToUpper()).$($DnsDomain.ToUpper())!!!  Now aborting nightly backup to external hard drive!!!" `
-Priority High -DNO onSuccess, onFailure -SmtpServer "mail.$DnsDomain"
break
}

As always – Use any tips, tricks, or scripts I post at your own risk.

HOWTO: Converting from BackupExec to #Veeam when using RDX drives

Ok – I’m completely done with Backup Exec when it comes to VMware.  I’ve been selling, supporting, certified on and even using Backup Exec for our own internal backups since it was Conner Backup Exec for Windows NT 3.1, way back in 1993.  Once upon a time, it was a great product – in fact it was the only product for backups that worked worth a damn.  But it’s reliability has dropped to nothing over the past 6 or 7 years.  Technical support has been off-shored and 99.9% of the time, if I am lucky enough to finally reach someone in technical support on the phone, I can’t understand a damn word they say due to their thick accent and shitty VOIP lines crossing the Pacific Ocean.  Today was the last straw with Backup Exec, their crappy bugs, and unreliable VMware backups.  So now it’s time to fully embrace the move to Veeam, which I’ve been considering for some time (note of disclosure – I am also a certified Veeam VMCE – v7, v8, & v9)

Several of my clients have single standalone ESXi hosts, an HPE StoreOnce appliance, a physical Windows Server 2012R2 with a RDX drive or two (for offline backups), and both Backup Exec and Veeam loaded on that Windows server.  Oh – and many, many, many RDX cartridges that have months of rotated backups on them that are all three quarters full.  I can’t just erase all these cartridges in one swoop and use them for Veeam backups.  And I certainly don’t want to have to log into the clients’ servers everyday to manually delete the old Backup Exec folders off the RDX (as they come up in rotation) so that there is enough room for the nightly Veeam backup.  And finally, even though I’m dumping Backup Exec for my VMware backups, I still need to use Backup Exec to backup the 2012R2 physical instance to the same RDX cartridge that Veeam is going to use (atleast until Veeam releases their next project).  So what do I do?

A little PowerShell scripting to the rescue – that is what I am going do!

After going through a sampling of several RDX cartridges at several different client sites, I’ve determined that when Backup Exec runs with GRT enabled it dumps those backed up VMs in IMGxxxxxx folders on the root of the RDX drive (including the VMDKs).  I also discovered (or at least in the environments that I’ve setup) that GRT enabled application backups (not VMs, but rather SQL, AD, Exchange) will also be in an IMG folder with either a file called ntds.dit or edb.chk, and sometimes both!  In my case, my 2012R2 server has SQL and AD on it, so I want to be careful not to delete IMG folders that potentially contain my SQL and AD backups (which could screw Backup Exec up even more than normal when it uses that cartridge again for the 2012R2 server).

In the end, I setup the RDX drive as a new rotated drive repository in Veeam (prior to this Veeam only backed up to the HPE StoreOnce).  I then create a new Veeam job that did active fulls to the RDX drive every night (with a restore points to keep of 1).  In the job’s Advanced Settings menu, I added a pre-run script that runs C:\Windows\Remove_BackupExec_IMG_Folders.cmd.  This script in turn launches a PowerShell script that deletes all the IMGxxxxxx folders off the RDX drive except IMG folders that contain either ntds.dit or edb.chk.

**NOTE – the following deletes data from your backup cartridges. Use any tips, tricks, or scripts I post at your own risk.  I accept zero liability and responsibility if you use these scripts!!!**

Here is the contents of my batch file.

rem start notepad++ "C:\Windows\Remove_BackupExec_IMG_Folders.cmd"
PowerShell -NoProfile -ExecutionPolicy Bypass -Command "& {Start-Process PowerShell -ArgumentList '-NoProfile -ExecutionPolicy Bypass -File ""C:\Windows\Remove_BackupExec_IMG_Folders.ps1""' -Verb RunAs}"
exit/b

Here is the contents of the PowerShell script to remove the IMGxxxxxx folders (adjust the drive letter accordingly)

# start notepad++ "C:\Windows\Remove_BackupExec_IMG_Folders.ps1"
foreach ($i in Get-ChildItem R:\IMG*)
{if ((test-path "$i\ntds.dit") -eq $False -and (test-path "$i\edb.chk") -eq $False) {Remove-Item $i -force -recurse -confirm:$false}}

But wait! There is more!

Because I am still going to have to suffer with Backup Exec a while longer to backup my 2012R2 server, I need to make sure my nightly Backup Exec job doesn’t eject the RDX cartridge on me before Veeam finishes it’s RDX job.  To ensure this, I disabled the scheduled RDX jobs on my Backup Exec server.  Fortunately, Backup Exec includes a PowerShell module called BEMCLI.  So I wrote a second set of scripts as it was simply a matter of starting PowerShell from a script, importing the module, and starting the job.  So this time my scripts are a post-job script to start the Backup Exec job only after the Veeam job completes.

2017-01-11-16-33-43-snagit-0002

Here is the batch file to launch PowerShell.

rem start notepad++ "C:\Windows\Start_BE_UTIL01_RDX_JOB.cmd"
PowerShell -NoProfile -ExecutionPolicy Bypass -Command "& {Start-Process PowerShell -ArgumentList '-NoProfile -ExecutionPolicy Bypass -File ""C:\Windows\Start_BE_UTIL01_RDX_JOB.ps1""' -Verb RunAs}"
exit/b

And here is the PowerShell script to start the Backup Exec job called “23:10 UTIL01 RDX-Full”.

# start notepad++ "C:\Windows\Start_BE_UTIL01_RDX_JOB.ps1"
Import-Module BEMCLI
Get-BEJob -Name "23:10 UTIL01 RDX-Full" | Start-BEJob -confirm:$false

Now when my Veeam backup job to RDX starts, it deletes all the IMGxxxxxx folders off the RDX drive (unless those folders contain either ntds.dit or edb.chk), and when it completes, it starts the remaining Backup Exec job, which ultimately ejects the RDX cartridge when it completes.

As always – Use any tips, tricks, or scripts I post at your own risk.

HOWTO: Silently remove old VMware vCenter 5.x apps/tools and install the newest 6.x ones

It appears that VMware has finally figured out how to make vSphere 6 stable, which means it is finally time for my team to start migrating our clients off vSphere 5.5 and onto vSphere 6.    Upgrading a vSphere host takes all of 60 seconds with esxcli followed by a reboot of the host.  Upgrading all the apps and tools to manage the vSphere hosts however can take hours if doing it manually across all the machines in a domain though.

Like many of you (I’m sure), we generally have the various VMware apps and tools such as VIClient, PowerCLI, VMRC, Client Integration Plugin, and Update Manager client installed on multiple machines throughout the client’s computer system.  It’s time consuming and a real pain in the butt to go into Add/Remove programs and manually uninstall all the old 5.x tools and then manually install all the new 6.0 tools on each of these machines.  So, after a bit of testing and troubleshooting, I’ve come up with a series of one liners to cut and paste into an administrative command prompt to do all the time consuming pain the butt work without actually doing any of the work myself… Using these scripts, I can generally remove all the old 4.x and 5.x software and install all the new 6.0 software in less than 5 minutes per machine.

And I use our inventory and software management system to determine which machines have 5.x apps installed on them before I ever begin so I can target just the machines I need to without wasting time.  So basically once I have all ESXi hosts upgraded to version 6, I use Remote Desktop Connection Manager to connect to each machine I have identified as having 5.x apps, open an Administrator command prompt and cut and paste all my command lines in (both uninstall and install).  Once that machine is cranking away, I move onto the next machine and start the same process over again, and then on to the next machine.  Generally the first machine is completed before I get the last machine even started.  Then it is just a matter of verifying the apps work as expected…

So first we want remove all existing VMware apps on the target machine except VMware Tools and VMware Update Manager (server, not client).  You should be able to cut and paste all 7 of these command lines into the administrative command prompt at the same time and they will run one after the other, silently uninstalling any installed application on the machine with a name that matches the search parameters. The 7th line will open Add/Remove programs for you so you can manually verify everything has been removed before continuing.

Important – make sure there are no opened/running browsers on the machine and that none of the VMware apps are opened (very important if you are doing this on a Windows server that allows Remote Desktop for Administration and another admin is logged into it at the same time!!!)

**Note – if you are still using a Windows based vCenter server – it likely wouldn’t be too wise to run these uninstall commands on the vCenter server – consider yourself warned**

start /wait wmic product where "name like 'vmware c%%'" call uninstall
start /wait wmic product where "name like 'vmware r%%'" call uninstall
start /wait wmic product where "name like 'vmware vix%%'" call uninstall
start /wait wmic product where "name like 'vmware vsphere c%%'" call uninstall
start /wait wmic product where "name like 'vmware vsphere p%%'" call uninstall
start /wait wmic product where "name like 'vmware vsphere update manager c%%'" call uninstall
start appwiz.cpl

So now we all our old version 5.x tools and apps removed from our management stations, so we can go ahead now and silently deploy our new version 6.x apps.  Again, you should be able to paste these 6 lines all at once into a command prompt and they will run sequentially and install the VIClient, VMware Remote Console, vSphere CLI, vSphere PowerCLI, and the Update Manager client.

 

start /wait \\SERVER\SETUP\VMWARE\ESXi60u02\VMware-viclient.exe /q /s /w /L1033 /v" /qr"
start /wait msiexec /qb- /i \\SERVER\SETUP\VMWARE\ESXi60u02\VMware-VMRC-9.0.0-4288332.msi EULAS_AGREED=1 AUTOSOFTWAREUPDATE=0 DATACOLLECTION=0
start /wait \\SERVER\SETUP\VMWARE\ESXi60u02\VMware-vSphere-CLI-6.0.0-3561779.exe  /s /v/qn
start /wait \\SERVER\SETUP\VMWARE\ESXi60u02\VMware-PowerCLI-6.3.0-3737840.exe  /s /v/qn 
start /wait \\SERVER\SETUP\VMWARE\ESXi60u02\VMware-UMClient.exe  /s /v/qn 
start appwiz.cpl

 

Add/Remove Programs should once again automatically open for you to manually verify that everything has installed correctly.

As always – Use any tips, tricks, or scripts I post at your own risk.

REVISED – HOWTO: Grab the all file download links on a HP Inc driver download page and wget them

About 3 months ago, I posted how to grab all the download links on a HP Inc driver download page and wget them.  After some messing around, I decided to take my previously posted instructions (which you can find here) and turn them into a pair of macros for Notepad++ to save some manual labor.  Below is the content of the two macros.  I also assigned ALT+F12 to the first macro, and ALT+F10 to the second macro.  And finally I assigned ALT+F11 to the Hex to Ascii plugin.

So now all I need to do is open Notepad++, paste the HP Inc’s driver download page’s source into Notepad++ and hit ALT+F12, ALT+F11, and then ALT+F10 to get my wget links.

To use these two macro’s, you’ll need to edit %AppData%\Notepad++\shortcuts.xml and insert these two macros, then save shortcuts.xml.  You also need to close Notepad++ and reopen it before using the macros.

Macro #1 – HP Inc Download Source Cleanup Part 1 – ALT+F12

<Macro name="HP Inc Download Source Cleanup Part 1" Ctrl="no" Alt="yes" Shift="no" Key="123">
<Action type="3" message="1700" wParam="0" lParam="0" sParam="" />
<Action type="3" message="1601" wParam="0" lParam="0" sParam="#" />
<Action type="3" message="1625" wParam="0" lParam="1" sParam="" />
<Action type="3" message="1602" wParam="0" lParam="0" sParam="" />
<Action type="3" message="1702" wParam="0" lParam="768" sParam="" />
<Action type="3" message="1701" wParam="0" lParam="1609" sParam="" />
<Action type="3" message="1700" wParam="0" lParam="0" sParam="" />
<Action type="3" message="1601" wParam="0" lParam="0" sParam="\t" />
<Action type="3" message="1625" wParam="0" lParam="1" sParam="" />
<Action type="3" message="1602" wParam="0" lParam="0" sParam="" />
<Action type="3" message="1702" wParam="0" lParam="768" sParam="" />
<Action type="3" message="1701" wParam="0" lParam="1609" sParam="" />
<Action type="3" message="1700" wParam="0" lParam="0" sParam="" />
<Action type="3" message="1601" wParam="0" lParam="0" sParam='&apos;&quot;&gt;\r\n' />
<Action type="3" message="1625" wParam="0" lParam="1" sParam="" />
<Action type="3" message="1602" wParam="0" lParam="0" sParam="\r\n" />
<Action type="3" message="1702" wParam="0" lParam="768" sParam="" />
<Action type="3" message="1701" wParam="0" lParam="1609" sParam="" />
<Action type="3" message="1700" wParam="0" lParam="0" sParam="" />
<Action type="3" message="1601" wParam="0" lParam="0" sParam='&quot;&gt;\r\n' />
<Action type="3" message="1625" wParam="0" lParam="1" sParam="" />
<Action type="3" message="1602" wParam="0" lParam="0" sParam="\r\n" />
<Action type="3" message="1702" wParam="0" lParam="768" sParam="" />
<Action type="3" message="1701" wParam="0" lParam="1609" sParam="" />
<Action type="3" message="1700" wParam="0" lParam="0" sParam="" />
<Action type="3" message="1601" wParam="0" lParam="0" sParam="obtainSoftware?url=" />
<Action type="3" message="1625" wParam="0" lParam="1" sParam="" />
<Action type="3" message="1602" wParam="0" lParam="0" sParam="\r\n###" />
<Action type="3" message="1702" wParam="0" lParam="768" sParam="" />
<Action type="3" message="1701" wParam="0" lParam="1609" sParam="" />
<Action type="0" message="2013" wParam="0" lParam="0" sParam="" />
<Action type="2" message="0" wParam="42043" lParam="0" sParam="" />
<Action type="2" message="0" wParam="42056" lParam="0" sParam="" />
<Action type="2" message="0" wParam="42059" lParam="0" sParam="" />
<Action type="3" message="1700" wParam="0" lParam="0" sParam="" />
<Action type="3" message="1601" wParam="0" lParam="0" sParam="^###" />
<Action type="3" message="1625" wParam="0" lParam="2" sParam="" />
<Action type="3" message="1702" wParam="0" lParam="784" sParam="" />
<Action type="3" message="1701" wParam="0" lParam="1615" sParam="" />
<Action type="2" message="0" wParam="43051" lParam="0" sParam="" />
<Action type="3" message="1700" wParam="0" lParam="0" sParam="" />
<Action type="3" message="1601" wParam="0" lParam="0" sParam="###" />
<Action type="3" message="1625" wParam="0" lParam="1" sParam="" />
<Action type="3" message="1602" wParam="0" lParam="0" sParam="" />
<Action type="3" message="1702" wParam="0" lParam="768" sParam="" />
<Action type="3" message="1701" wParam="0" lParam="1609" sParam="" />
<Action type="0" message="2013" wParam="0" lParam="0" sParam="" />
<Action type="2" message="0" wParam="43008" lParam="0" sParam="" />
</Macro>

Macro #2 – HP Inc Download Source Cleanup Part 2 – ALT+F10

<Macro name="HP Inc Download Source Cleanup Part 2" Ctrl="no" Alt="yes" Shift="no" Key="121">
<Action type="3" message="1700" wParam="0" lParam="0" sParam="" />
<Action type="3" message="1601" wParam="0" lParam="0" sParam="http:" />
<Action type="3" message="1625" wParam="0" lParam="1" sParam="" />
<Action type="3" message="1602" wParam="0" lParam="0" sParam="\r\nwget -c -N -T 20 -t 20 http:" />
<Action type="3" message="1702" wParam="0" lParam="896" sParam="" />
<Action type="3" message="1701" wParam="0" lParam="1609" sParam="" />
<Action type="0" message="2318" wParam="0" lParam="0" sParam="" />
<Action type="1" message="2170" wParam="0" lParam="0" sParam="" />
<Action type="1" message="2170" wParam="0" lParam="0" sParam="" />
</Macro>

As always – Use any tips, tricks, or scripts I post at your own risk.