HOWTO: Converting from BackupExec to #Veeam when using RDX drives

Ok – I’m completely done with Backup Exec when it comes to VMware.  I’ve been selling, supporting, certified on and even using Backup Exec for our own internal backups since it was Conner Backup Exec for Windows NT 3.1, way back in 1993.  Once upon a time, it was a great product – in fact it was the only product for backups that worked worth a damn.  But it’s reliability has dropped to nothing over the past 6 or 7 years.  Technical support has been off-shored and 99.9% of the time, if I am lucky enough to finally reach someone in technical support on the phone, I can’t understand a damn word they say due to their thick accent and shitty VOIP lines crossing the Pacific Ocean.  Today was the last straw with Backup Exec, their crappy bugs, and unreliable VMware backups.  So now it’s time to fully embrace the move to Veeam, which I’ve been considering for some time (note of disclosure – I am also a certified Veeam VMCE – v7, v8, & v9)

Several of my clients have single standalone ESXi hosts, an HPE StoreOnce appliance, a physical Windows Server 2012R2 with a RDX drive or two (for offline backups), and both Backup Exec and Veeam loaded on that Windows server.  Oh – and many, many, many RDX cartridges that have months of rotated backups on them that are all three quarters full.  I can’t just erase all these cartridges in one swoop and use them for Veeam backups.  And I certainly don’t want to have to log into the clients’ servers everyday to manually delete the old Backup Exec folders off the RDX (as they come up in rotation) so that there is enough room for the nightly Veeam backup.  And finally, even though I’m dumping Backup Exec for my VMware backups, I still need to use Backup Exec to backup the 2012R2 physical instance to the same RDX cartridge that Veeam is going to use (atleast until Veeam releases their next project).  So what do I do?

A little PowerShell scripting to the rescue – that is what I am going do!

After going through a sampling of several RDX cartridges at several different client sites, I’ve determined that when Backup Exec runs with GRT enabled it dumps those backed up VMs in IMGxxxxxx folders on the root of the RDX drive (including the VMDKs).  I also discovered (or at least in the environments that I’ve setup) that GRT enabled application backups (not VMs, but rather SQL, AD, Exchange) will also be in an IMG folder with either a file called ntds.dit or edb.chk, and sometimes both!  In my case, my 2012R2 server has SQL and AD on it, so I want to be careful not to delete IMG folders that potentially contain my SQL and AD backups (which could screw Backup Exec up even more than normal when it uses that cartridge again for the 2012R2 server).

In the end, I setup the RDX drive as a new rotated drive repository in Veeam (prior to this Veeam only backed up to the HPE StoreOnce).  I then create a new Veeam job that did active fulls to the RDX drive every night (with a restore points to keep of 1).  In the job’s Advanced Settings menu, I added a pre-run script that runs C:\Windows\Remove_BackupExec_IMG_Folders.cmd.  This script in turn launches a PowerShell script that deletes all the IMGxxxxxx folders off the RDX drive except IMG folders that contain either ntds.dit or edb.chk.

**NOTE – the following deletes data from your backup cartridges. Use any tips, tricks, or scripts I post at your own risk.  I accept zero liability and responsibility if you use these scripts!!!**

Here is the contents of my batch file.

rem start notepad++ "C:\Windows\Remove_BackupExec_IMG_Folders.cmd"
PowerShell -NoProfile -ExecutionPolicy Bypass -Command "& {Start-Process PowerShell -ArgumentList '-NoProfile -ExecutionPolicy Bypass -File ""C:\Windows\Remove_BackupExec_IMG_Folders.ps1""' -Verb RunAs}"
exit/b

Here is the contents of the PowerShell script to remove the IMGxxxxxx folders (adjust the drive letter accordingly)

# start notepad++ "C:\Windows\Remove_BackupExec_IMG_Folders.ps1"
foreach ($i in Get-ChildItem R:\IMG*)
{if ((test-path "$i\ntds.dit") -eq $False -and (test-path "$i\edb.chk") -eq $False) {Remove-Item $i -force -recurse -confirm:$false}}

But wait! There is more!

Because I am still going to have to suffer with Backup Exec a while longer to backup my 2012R2 server, I need to make sure my nightly Backup Exec job doesn’t eject the RDX cartridge on me before Veeam finishes it’s RDX job.  To ensure this, I disabled the scheduled RDX jobs on my Backup Exec server.  Fortunately, Backup Exec includes a PowerShell module called BEMCLI.  So I wrote a second set of scripts as it was simply a matter of starting PowerShell from a script, importing the module, and starting the job.  So this time my scripts are a post-job script to start the Backup Exec job only after the Veeam job completes.

2017-01-11-16-33-43-snagit-0002

Here is the batch file to launch PowerShell.

rem start notepad++ "C:\Windows\Start_BE_UTIL01_RDX_JOB.cmd"
PowerShell -NoProfile -ExecutionPolicy Bypass -Command "& {Start-Process PowerShell -ArgumentList '-NoProfile -ExecutionPolicy Bypass -File ""C:\Windows\Start_BE_UTIL01_RDX_JOB.ps1""' -Verb RunAs}"
exit/b

And here is the PowerShell script to start the Backup Exec job called “23:10 UTIL01 RDX-Full”.

# start notepad++ "C:\Windows\Start_BE_UTIL01_RDX_JOB.ps1"
Import-Module BEMCLI
Get-BEJob -Name "23:10 UTIL01 RDX-Full" | Start-BEJob -confirm:$false

Now when my Veeam backup job to RDX starts, it deletes all the IMGxxxxxx folders off the RDX drive (unless those folders contain either ntds.dit or edb.chk), and when it completes, it starts the remaining Backup Exec job, which ultimately ejects the RDX cartridge when it completes.

As always – Use any tips, tricks, or scripts I post at your own risk.

HOWTO: Silently remove old VMware vCenter 5.x apps/tools and install the newest 6.x ones

It appears that VMware has finally figured out how to make vSphere 6 stable, which means it is finally time for my team to start migrating our clients off vSphere 5.5 and onto vSphere 6.    Upgrading a vSphere host takes all of 60 seconds with esxcli followed by a reboot of the host.  Upgrading all the apps and tools to manage the vSphere hosts however can take hours if doing it manually across all the machines in a domain though.

Like many of you (I’m sure), we generally have the various VMware apps and tools such as VIClient, PowerCLI, VMRC, Client Integration Plugin, and Update Manager client installed on multiple machines throughout the client’s computer system.  It’s time consuming and a real pain in the butt to go into Add/Remove programs and manually uninstall all the old 5.x tools and then manually install all the new 6.0 tools on each of these machines.  So, after a bit of testing and troubleshooting, I’ve come up with a series of one liners to cut and paste into an administrative command prompt to do all the time consuming pain the butt work without actually doing any of the work myself… Using these scripts, I can generally remove all the old 4.x and 5.x software and install all the new 6.0 software in less than 5 minutes per machine.

And I use our inventory and software management system to determine which machines have 5.x apps installed on them before I ever begin so I can target just the machines I need to without wasting time.  So basically once I have all ESXi hosts upgraded to version 6, I use Remote Desktop Connection Manager to connect to each machine I have identified as having 5.x apps, open an Administrator command prompt and cut and paste all my command lines in (both uninstall and install).  Once that machine is cranking away, I move onto the next machine and start the same process over again, and then on to the next machine.  Generally the first machine is completed before I get the last machine even started.  Then it is just a matter of verifying the apps work as expected…

So first we want remove all existing VMware apps on the target machine except VMware Tools and VMware Update Manager (server, not client).  You should be able to cut and paste all 7 of these command lines into the administrative command prompt at the same time and they will run one after the other, silently uninstalling any installed application on the machine with a name that matches the search parameters. The 7th line will open Add/Remove programs for you so you can manually verify everything has been removed before continuing.

Important – make sure there are no opened/running browsers on the machine and that none of the VMware apps are opened (very important if you are doing this on a Windows server that allows Remote Desktop for Administration and another admin is logged into it at the same time!!!)

**Note – if you are still using a Windows based vCenter server – it likely wouldn’t be too wise to run these uninstall commands on the vCenter server – consider yourself warned**

start /wait wmic product where "name like 'vmware c%%'" call uninstall
start /wait wmic product where "name like 'vmware r%%'" call uninstall
start /wait wmic product where "name like 'vmware vix%%'" call uninstall
start /wait wmic product where "name like 'vmware vsphere c%%'" call uninstall
start /wait wmic product where "name like 'vmware vsphere p%%'" call uninstall
start /wait wmic product where "name like 'vmware vsphere update manager c%%'" call uninstall
start appwiz.cpl

So now we all our old version 5.x tools and apps removed from our management stations, so we can go ahead now and silently deploy our new version 6.x apps.  Again, you should be able to paste these 6 lines all at once into a command prompt and they will run sequentially and install the VIClient, VMware Remote Console, vSphere CLI, vSphere PowerCLI, and the Update Manager client.

 

start /wait \\SERVER\SETUP\VMWARE\ESXi60u02\VMware-viclient.exe /q /s /w /L1033 /v" /qr"
start /wait msiexec /qb- /i \\SERVER\SETUP\VMWARE\ESXi60u02\VMware-VMRC-9.0.0-4288332.msi EULAS_AGREED=1 AUTOSOFTWAREUPDATE=0 DATACOLLECTION=0
start /wait \\SERVER\SETUP\VMWARE\ESXi60u02\VMware-vSphere-CLI-6.0.0-3561779.exe  /s /v/qn
start /wait \\SERVER\SETUP\VMWARE\ESXi60u02\VMware-PowerCLI-6.3.0-3737840.exe  /s /v/qn 
start /wait \\SERVER\SETUP\VMWARE\ESXi60u02\VMware-UMClient.exe  /s /v/qn 
start appwiz.cpl

 

Add/Remove Programs should once again automatically open for you to manually verify that everything has installed correctly.

As always – Use any tips, tricks, or scripts I post at your own risk.

HOWTO: Turn on a HDD UID on a HPE Proliant in VMware with HPSSACLI

This morning we needed to replace a hard drive in a HPE Proliant running VMware ESXi at a remote site that had a PFA on it.  Unfortunately, while ILO is great at identifying the defective drive, it has no ability to enable the UID on the drive, and given that this unit is at a remote site, we had no way of knowing in advanced if the fault light was actually turn on for this drive before the HPE field engineering arrived to swap the drive.  So after digging through the help documentation, I found the necessary HPSSACLI command to enable the drive’s UID.

First, to get a list of all the physical drives in an ESXi host, SSH the host and run this command:

/opt/hp/hpssacli/bin/hpssacli ctrl slot=0 physicaldrive all show

This should output a list of all the drives in the system as shown below.

2016.05.19 - 10.14.13 - SNAGIT -  0005

Next, to enable the blue UID LED for 1 hour on port 2I, box 1, bay 2, run this command:

/opt/hp/hpssacli/bin/hpssacli ctrl slot=0 physicaldrive 2I:1:8 modify led=on duration=3600

The blue UID should now come on for 1 hour and then shut off on it’s own.  If you want want to manually shut if off before the 1 hour is up, run the same command again, but change the “led=on” to “led=off”.

As always – Use any tips, tricks, or scripts I post at your own risk.

HOWTO: Scheduled a standalone VMware ESXi Host Reboot via Powershell

We have several clients who have standalone VMware ESXi hosts (that are not part of any vCenter) without any option for vMotion or Storage vMotion.  This can make it difficult for us to keep those hosts current with patches, updates, and BIOS / firmware because it means we need to manually shut the hosts’ guest down, and then restart the host – none of which can be done during normal business hours – and I’m getting too old to work all night.

Fortunately, VMware provides us a way to use PowerShell to shutdown a ESXi host’s guest, and then force a reboot.  This means we can apply patches and updates late in the day to the ESXi host, then schedule the host to reboot early in the morning after the daily backup completes.  Then when we come into the office in the morning (usually an hour or two before the clients arrive at their offices), it is simply a matter of checking the host to ensure it is back up along with all it’s guests.

To schedule a standalone VMware Host reboot, the current VMware PowerCLI client needs to be installed on the machine that will be running the scheduled reboot.

Once the VMware PowerCLI is installed, you need to create 3 files:

  • C:\WINDOWS\VMWARE_ROOT.PWD – encrypted file that contains the root user’s password
  • C:\WINDOWS\VMWARE_HOST_REBOOT.CMD – the wrapper that will call PowerShell from TaskScheduler
  • C:\WINDOWS\VMWARE_HOST_REBOOT.PS1 – the actual PowerShell script that executes the reboot

To create the file C:\WINDOWS\VMWARE_ROOT.PWD, open PowerShell and run the following command:

read-host -assecurestring "Enter Password" | convertfrom-securestring | out-file C:\WINDOWS\VMWARE_ROOT.PWD

 

At the “Enter Password” prompt, enter the password of the root user account for the ESXi host you want to reboot.

You also need to set the PowerShell Execution Policy to support remote signed scripts such as C:\WINDOWS\VMWARE_HOST_REBOOT.PS1.  To do this, in PowerShell run the following command and select Yes when prompted:

Set-ExecutionPolicy RemoteSigned

We need to schedule a time for VMWARE_HOST_REBOOT.CMD to run.  I’ve set 4:15 am local time on March 22, 2015 in the example shown below, but you can adjust as required.  In an administrative command prompt, run this (***note – this will create the scheduled task to run as the currently logged in user***):

schtasks /create /tn "VMware Host Reboot" /tr C:\WINDOWS\VMWARE_HOST_REBOOT.CMD /sc once /st 04:15:00 /sd 03/22/2015 /rp "*" /ru "%userdomain%\%username%"

Now we need to create C:\WINDOWS\VMWARE_HOST_REBOOT.CMD, which is the batch file task scheduler uses to launch our PowerShell script.

rem --- begin cut and paste of notepad C:\WINDOWS\VMWARE_HOST_REBOOT.CMD
@echo off
C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -ExecutionPolicy RemoteSigned -noprofile -File C:\WINDOWS\VMWARE_HOST_REBOOT.PS1
exit /b
rem --- end cut and paste of C:\WINDOWS\VMWARE_HOST_REBOOT.CMD ---

Lastly, we need to create C:\WINDOWS\VMWARE_HOST_REBOOT.PS1, adjusting the variable for $server to the host you wish to reboot (all variables are all defined at the top of the script) and adjust wait time ($waittime) before force rebooting after you issue a graceful gust shutdown command.

###--- begin cut and paste of notepad C:\WINDOWS\VMWARE_HOST_REBOOT.PS1
### VMWARE_HOST_REBOOT.PS1
### @deancolpitts – http://blog.jbgeek.net
### 2015.03.20

### This script will attempt to perform a graceful VM restart via the VMware Tools inside the guest.
### Variables - please only adjust server, user, and waittime. Any other variables should not be touched.
### Server is the vCenter server or ESXi host's FQDN, while user is the vCenter user or ESXi user account.

$server = "VMWARE.FQDN.DOMAIN_OR_IPADDRESS"
$user = "root"
$waittime = "300"

$credentialFile = "C:\WINDOWS\VMWARE_ROOT.PWD"
$pass = cat $credentialFile | convertto-securestring
$credentials = new-object -typename System.Management.Automation.PSCredential -argumentlist $user,$pass

add-pssnapin VMware.VimAutomation.Core -ErrorAction SilentlyContinue -WarningAction SilentlyContinue | Out-Null
if ( $DefaultVIServers.Length -lt 1 )
{
Connect-VIServer -Server $server -Protocol https -credential $credentials -WarningAction SilentlyContinue | Out-Null
}

Get-VM | Shutdown-VMGuest -confirm:$false -WarningAction SilentlyContinue

### Wait x number of seconds for all the VM's to gracefully shutdown before a forced kill occurs
Start-Sleep -s $waittime

Restart-VMHost -VMHost $server -force -confirm:$false

###--- end cut and paste of C:\WINDOWS\VMWARE_HOST_REBOOT.PS1 ---

All that is left do now is wait for C:\WINDOWS\VMWARE_HOST_REBOOT.CMD to run at your scheduled time.

As always – Use any tips, tricks, or scripts I post at your own risk.

HOWTO: Schedule Daily Netscaler VPX Reboots via Powershell

We often utilize Citrix’s NetScaler VPX running on VMware ESXi 5.5 to allow our clients to securely connect to their Citrix infrastructure from outside the firewall.  For the most part – it works well.  Unfortunately though, our experience has taught us that occasionally NSVPX goes all fubar on it’s own after a few days of running and stops processing connection requests once the user logs in.  A simple reboot of the NSVPX VM usually resolves the user’s connectivity issues..

To combat this issue, I wrote a Powershell script that we run as a daily scheduled task on our management server to have vCenter automatically restart the machine once a day.  You could easily modify this script to reboot any VM you want though.

To configure daily VM rebooting, the current VMware PowerCLI client needs to be installed on the machine that will be running the scheduled reboot.  Once the VMware PowerCLI is installed, you need to create 3 files on the management machine:

  1. daily_nsvpx_reboot.cmd – which is the wrapper that will call PowerShell from TaskScheduler (see below in for cut and paste of the file contents)
  2. daily_nsvpx_reboot.ps1 – which is the actual PowerShell script that executes the reboot (see below for cut and paste of the file contents)
  3. daily_nsvpx_reboot.pwd – which is an encrypted file that contains the vCenter user’s password

To create the file daily_nsvpx_reboot.pwd, open PowerShell and run the following command:

read-host -assecurestring "Enter Password" | convertfrom-securestring | out-file c:\windows\daily_nsvpx_reboot.pwd

At the “Enter Password” prompt, enter the password of the user account you will be using that has rights in vCenter or the ESXi host to perform VM restarts.

You may also need to set the PowerShell Execution Policy to support remote signed scripts such as daily_nsvpx_reboot.ps1.  To do this, open PowerShell and run the following command and select Yes when prompted:

Set-ExecutionPolicy RemoteSigned

After creating daily_nsvpx_reboot.cmd and daily_nsvpx_reboot.ps1 (see below for file contents of these two files), edit daily_nsvpx_reboot.ps1 and adjust the variables for $server, $user, and $vm2reboot to fit your environment (these three variables are all defined at the top of the script).

Lastly, you need to schedule daily_nsvpx_reboot.cmd to run daily.  I’ve set 4:15 am local time in the example shown below, but you can adjust as required.  To schedule the task, open an Administrative command prompt and run the following command (adjust domain\username to be the same user account that has rights in vCenter or the ESXi host to perform VM restarts):

schtasks /create /tn "Daily NSVPX Reboot" /tr C:\WINDOWS\DAILY_NSVPX_REBOOT.CMD /sc daily /st 04:15:00 /rp "*" /ru "domain\username"

All that is left do now is test run daily_nsvpx_reboot.cmd and see that it runs and reboots the NSVPX.  If you are monitoring via ProcExp or TaskManager on the management machine, you should note low CPU usage followed by several spikes up to 50% (it is single threaded), and you should be able to see in the NSVPX console via vCenter when it reboots.

And as always – Use any tips, tricks, or scripts I post at your own risk.


daily_nsvpx_reboot.cmd – file contents

rem — begin cut and paste of notepad c:\windows\daily_nsvpx_reboot.cmd
@echo off
C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -ExecutionPolicy RemoteSigned -noprofile -File C:\Windows\daily_nsvpx_reboot.ps1
exit /b
rem — end cut and paste of c:\windows\daily_nsvpx_reboot.cmd —

daily_nsvpx_reboot.ps1 – file contents

    ###— begin cut and paste of notepad c:\windows\daily_nsvpx_reboot.ps1

    ### Daily_nsvpx_reboot.ps1
    ### @deancolpitts – http://blog.jbgeek.net
    ### 2015.01.02
    ### This script will attempt to perform a graceful VM restart via the VMware Tools inside the guest.

    ### Variables – please only adjust server, user, and vm2reboot.  Any other variables should not be touched.
    ### Server is the vCenter server or ESXi host’s FQDN, while user is the vCenter user or ESXi user account.
    ### if any smtp variables present, they should be self-explanatory.

    $server = “vcenter.domain.fqdn”
    $user = “vcenter_username”
    $vm2reboot = “nsvpx”

    ### Read the encrypted user password from “c:\windows\daily_nsvpx_reboot.pwd”
    ### Use the following commented out PowerShell command to manually create a new credentials store.
    ### Enter the user’s password when prompted while running the read-host command
    ### read-host -assecurestring “Enter Password” | convertfrom-securestring | out-file c:\windows\daily_nsvpx_reboot.pwd

    $credentialFile = “c:\windows\daily_nsvpx_reboot.pwd”
    $pass = cat $credentialFile | convertto-securestring
    $credentials = new-object -typename System.Management.Automation.PSCredential -argumentlist $user,$pass

    add-pssnapin VMware.VimAutomation.Core -ErrorAction SilentlyContinue -WarningAction SilentlyContinue | Out-Null

    if ( $DefaultVIServers.Length -lt 1 )
    {
    Connect-VIServer -Server $server -Protocol https -credential $credentials -WarningAction SilentlyContinue | Out-Null
    }

    Restart-VM -VM $vm2reboot -RunAsync -Confirm:$false

    ###— end cut and paste of c:\windows\daily_nsvpx_reboot.ps1 —