HOWTO: Restore a dead or deleted vCenter server from an @HPE_Simplivity backup

This morning was vCenter update day for me. I had 15 customer vCenter instances that all needed upgraded from 7.0.3.01000 to 7.0.3.01100, so I grabbed a cup of coffee and got started. 14 of the 15 completed with out a hitch, but there is always one! This one vCenter server failed to install the patch, leaving me with a dead vCenter. And this particular vCenter is residing on an HPE Simplivity cluster.

In case you didn’t know, Simplivity has it’s own built in backup and restore mechanism, which is generally accessed via the vCenter client. Which is cool, until your vCenter is dead, and you need to restore your vCenter from those backups, which is done via vCenter (that same dead vCenter you are attempting to restore). Then what do you do? HPE’s documentation on this isn’t super clear. I’d been down this same road earlier this year, so I had already trudged through the framework of what to do once, but I actually hadn’t written down. So this time – not only am I documenting it, I’m sharing it with you!

And as always before I begin:

Use any tips, tricks, or scripts I post at your own risk.

Open Putty and ssh one of the OmniStackVC VMs.

Login as svtcli / yourpassword (this is your emergency password)

Find the available backups:    svt-backup-show --emergency

The first column shows the Datastore name.  The second column is the VM name.  The third column is the backup name and will generally correspond to the backup time. It’s possible to do more granular searches with svt-backup-show. Use --help to get the parameters if you need to narrow down the results.

If the VM has been deleted, then it’s name will show as “VMNAME [Deleted] YYYY-MM-DDTHH:MM:SS+OFFSET” in this list (i.e. “VCENTER01 [DELETED 2022-12-10T13:20:34+0000]” in my example below)

**Note** Your text may be wrapped in Putty – I recommend copying and pasting the text out of Putty into Notepad++ or some other editor for easier reading.

To restore the VM, you’ll need to know the Datastore, the Object, and Backup Name (which is the time of the backup) you are restoring.  The syntax for a restore is this:

svt-backup-restore --datastore “Datastore --vm “Object --backup “Backup Name--emergency --force

So in my case, it was:  svt-backup-restore --datastore “SVT-DS02--vm “VCENTER01 [DELETED 2022-12-10T13:20:34+0000]--backup “2022-12-10T07:00:00-04:00--emergency –force

If everything worked correctly, you should see a Task Complete. The VM will then be restored into a new folder on the original datastore.

**Note** It may take a minute or two before the restored VM actually appears on the datastore. Be patient! If you simply hit the up arrow and hit enter again to run the restore again, you’ll end up with another copy!

If your original VM has been deleted, then you can safely rename this folder as required to match the original VM’s name.  I’m taking these screenshots after the fact, so the existing VCENTER01 shown below is the one I restored earlier this morning (and is now back into production) which inspired this writing – the VCENTER01-restored-blahblahblah is the one I just restored in the screenshots above for my documentation.

Now you can log into the WebUI of one of your ESXi nodes as root, register the recovered vCenter, and power it on. To register the VM, right click Virtual Machines, select “Create/Register VM”, select “register an existing Virtual Machine”, navigate to the datastore and select the restored .vmx file.

**Note** I’m not particularly happy with the editor in WordPress anymore… If anyone knows how I can write these posts in Outlook or Word and then copy and paste (including the formatting) into WordPress, please let me now.

HOWTO: Using #PowerShell to ensure a #Veeam USB Repository always has the correct drive letter

Some time ago I had a customer who switched completely to Veeam from Backup Exec (yeah baby!!!).  Offsite replication of any sort was out of the question as the customer simply couldn’t get the necessary bandwidth from any of the ISPs that serviced the area at anything that even approached an unreasonable price.  Instead we opted for a HPE StoreOnce for onsite and some sort of removable drive system for daily offsite.  The customer insisted on using Western Digital My Books (USB3) due to their capacity (6TB) and semi-ruggedness for the offline repo.  The drives are rotated out each day by a staff member for a new drive and they send the previous night’s drive offsite.

The My Books were originally setup using F: as the drive letter.  This would work great until someone plugged a USB key into the server and it was auto-assigned F: by Windows 2012R2, or if someone logged in who had a drive mapping of F:.  And then other days, Windows would randomly assign a drive letter other than F: even though F: was available.  At this point, Veeam would puke because it couldn’t find the repo anymore.  And occasionally the customer would forget to swap the drive for a fresh one (or even correct drive some days), which meant there wouldn’t be enough free space on the repo to complete that night’s backup (about 4.9TB is processed daily).

To work around all this, I ended up writing a PowerShell script to always map the My Book to Z:, clean the previous backups off of it (if any were found, or if the wrong drive with a different backup set was plugged in), and start the actual backup job.  And if the script can’t find the My Book or free disk capacity is less than 95% after attempting cleanup, it will abort the backup and send an email warning to the support team that the backup was aborted.  Rather than have Veeam schedule the job to run, I use Windows Task Scheduler instead to run the script, then use the Veeam PowerShell module to start the backup job from inside the script after I’ve verified Z: is present.

This script assumes that every rotated My Book drive has been formatted and has a volume label containing “WD MY BOOK”.  It also assumes the SMTP server is called mail dot whatever the DNS domain name of the machine running the script is (i.e. my AD is jbgeek.net, so mail.jbgeek.net).  All of my client sites have a cname for their Exchange server called “mail” created in their AD DNS zone – which means I can cut and paste the same script without modification from one client site to another, which cuts down the chance of an editing error, and makes it quicker to deploy new scripts when necessary for my team.

You will need to adjust the “Get-ChildItem -Path” (lines 21 to 26) directories to fit your environment, along with the $SendEmailTo variable (line 7) and the “Start-VBRJob -Job” job name (line 41).  Line 28 defines the cutoff point for disk capacity – if the available disk space is less than 95% of the drive capacity, the backup aborts.  Line 1 is commented it out – I generally always start my scripts with # start notepad++ script.filename so that I can just cut and paste it into a command prompt to create my script file – that way as I on-board each new customer, their sites are configured the same as existing sites.  Again, it cuts down the chances of an error and makes it quicker for my guys to deploy new scripts.

**NOTE – the following deletes data from your backup cartridges. Use any tips, tricks, or scripts I post at your own risk.  I accept zero liability and responsibility if you use these scripts!!!**

Here is the command line to create the scheduled task to run as the logged in user at 10pm each weekday (it will prompt you for your password when you run it in a command prompt):

schtasks /create /tn "Weekday Veeam USB Drive Backup" /tr "\"C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe\" -ExecutionPolicy RemoteSigned -noprofile -File  C:\Windows\Run_External_Drive_Veeam_Backup.ps1" /sc daily /st 22:00:00  /rp "*" /ru "%userdomain%\%username%"

Here is the Powershell script (updated 2017.01.17 to fix an error):

# start notepad++ C:\Windows\Run_External_Drive_Veeam_Backup.ps1

add-pssnapin Veeam*

$DnsDomain = Get-WmiObject -Class Win32_NTDomain -Filter "DSDirectoryServiceFlag='True'" | Select -ExpandProperty DnsForestName
$ThisComputerName = Get-WmiObject -Class Win32_ComputerSystem | Select -ExpandProperty Name
$SupportEmailAddress = "support@$DnsDomain"

# Get-WmiObject Win32_Volume | Where-Object {$_.DriveLetter -eq $null}
$drives = Get-WmiObject -Class win32_volume | Where-Object {$_.Label -match "WD MY BOOK"}
$i = 0
Foreach($drive in $drives)
{
#Set letter
$DriveLetter = "Z:"
Set-WmiInstance -input $drive -Arguments @{DriveLetter="$DriveLetter"}
$i++
}

if (Test-Path -path Z:\){
Get-ChildItem -Path "Z:\Veeam\Backups\All VMs - Z Drive" -Include *.vbm -recurse | foreach { $_.Delete()}
Get-ChildItem -Path "Z:\Veeam\Backups\All VMs - Z Drive" -Include *.vib -recurse | foreach { $_.Delete()}
Get-ChildItem -Path "Z:\Veeam\Backups\All VMs - Z Drive" -Include *.vbk -recurse | foreach { $_.Delete()}
Get-ChildItem -Path "Z:\Veeam\Backups\All VMs" -Include *.vbm -recurse | foreach { $_.Delete()}
Get-ChildItem -Path "Z:\Veeam\Backups\All VMs" -Include *.vib -recurse | foreach { $_.Delete()}
Get-ChildItem -Path "Z:\Veeam\Backups\All VMs" -Include *.vbk -recurse | foreach { $_.Delete()}
$externaldrivelabel = Get-WmiObject -Class win32_volume | Where-Object {$_.Label -match "WD MY BOOK"} | Select -ExpandProperty label
Get-WmiObject -Class win32_volume | Where-Object {$_.Label -match "WD MY BOOK"} | % { if (($_.FreeSpace/$_.Capacity) -le '0.95' )
{Send-MailMessage -From "$($ThisComputerName.ToUpper())@$($DnsDomain.ToUpper())" -To "$SupportEmailAddress" `
-Subject "Backup Aborted due to low space on $($externaldrivelabel)." `
-Body  "Aborting nightly Veeam backup to $($externaldrivelabel) (Z:) on $($ThisComputerName.ToUpper()).$($DnsDomain.ToUpper()) due to low disk space." `
-Priority High -DNO onSuccess, onFailure -SmtpServer "mail.$DnsDomain"
break}
}
}
if (Test-Path -path Z:\){
Send-MailMessage -From "$($ThisComputerName.ToUpper())@$($DnsDomain.ToUpper())" -To "$SupportEmailAddress" `
-Subject "Now starting nightly Veeam backup to $($externaldrivelabel)." `
-Body  "Now starting nightly Veeam backup to $($externaldrivelabel) (Z:) on $($ThisComputerName.ToUpper()).$($DnsDomain.ToUpper())." `
-Priority High -DNO onSuccess, onFailure -SmtpServer "mail.$DnsDomain"
Start-VBRJob -Job "All VMs - Z Drive" -FullBackup -RunAsync
break
}else{
Send-MailMessage -From "$($ThisComputerName.ToUpper())@$($DnsDomain.ToUpper())" -To "$SupportEmailAddress" `
-Subject "Error:  No external backup drive (Z:) found on $($ThisComputerName.ToUpper()).$($DnsDomain.ToUpper())!!!" `
-Body  "Error:  Cannot find or mount the external backup drive (Z:) on $($ThisComputerName.ToUpper()).$($DnsDomain.ToUpper())!!!  Now aborting nightly backup to external hard drive!!!" `
-Priority High -DNO onSuccess, onFailure -SmtpServer "mail.$DnsDomain"
break
}

As always – Use any tips, tricks, or scripts I post at your own risk.