HOWTO: Send “Shift + F10” to a remote #VMware / #vPro console over #RDP

We typically do a lot of our work at client locations over RDP to a server that resides on-prem in the client’s data center.  We then use that on-prem server as a jump server to manage other systems that reside on-prem and for the most part it works well except for the occasion keyboard combo press that just won’t go through to the client machine we are remoted into via the RDP jump server.  This is particularly a problem is when we are deploying Windows images to machines, which we do either via Intel’s vPro KVM controls (if it is physical) or via VMware Console (if it is virtual).  Occasionally the Windows machine will fail to boot during the specialize phase of sysprep, and we need to troubleshoot the issue.

If we were physically sitting in front of the machine, the process would be pretty simple – hit Shift + F10 and a command prompt pops open, and from there you navigate to C:\Windows\Panther and access the sysprep logs using Notepad.  But in our case, because we are utilizing a jump server via RDP to access the console (either vPro or VMware), Shift + F10 is being intercept by the jump server and not passed on to the vPro KVM session or the VMware Console, which means we can’t get to a command prompt to start troubleshooting.  When this happens, we need to disconnect from the RDP session, and use either RealVNC Plus (for the vPro console) or the VMware client directly from our local machines over VPN, which in some cases is deathly slow at best.

After getting stuck the other day having to troubleshoot a sysprep error over VPN using vPro instead of RDP using vPro, I decided there had to be a way to script a hotkey to send Shift + F10 to the console via RDP.  Unfortunately, I didn’t find anything readily available, so I scripted my own using AutoIt, which I then compiled into an .exe and digitally signed with my code signing certificate.

Basically the script searches for a window that has a title of “Intel(r) AMT KVM – VNC Viewer Plus” or a window that has a title that contains ” on ” (as in Windows7 on ESXiHost” and makes the first instance it finds the active window, then sends a Shift + F10 to the window.

Now when I have to send a Shift + F10 to a remote console during troubleshooting, I simply run the correct executable on our RDP jump servers and up comes the command prompt in the remote console!

Below are the two AutoIt scripts and further down are the two compiled .exe files.

As always – Use any tips, tricks, or scripts I post at your own risk.

Code for Intel(r) AMT KVM – VNC Viewer Plus:

$Title = "Intel(r) AMT KVM - VNC Viewer Plus"
WinWait ($Title)
WinActivate ($Title)
Send ("+{F10}")
Sleep (100)
Exit

Code for VMware Console:

Opt("MouseCoordMode", 0)
Opt("WinTitleMatchMode", 2)
$Title = "[REGEXPTITLE:(?i)(.* on .*)]"

WinWait ($Title)
WinActivate ($Title)

MouseClick ( "right",9,88,1 )
Send ("+{F10}")
Sleep (100)
Exit

Already compiled and digitally signed AutoIT executables:

SendShiftF10toVMware
SendShiftF10tovPro

HOWTO: Fix Windows Server 2016 BSOD Stop 0x00000133 after a failed Cumulative Update installation (#WindowsServer2016 #BSOD #Microsoft)

This morning I logged into my HPE Proliant DL60 which is running Windows Server 2016 (1607) and noticed it wanted to install KB4093119, which is the “2018-04 Cumulative Update for Windows Server 2016 for x64-based Systems (KB4093119)”.  When I was done doing what I had originally logged in for, I told the update to install and reboot.  After many reboots (I don’t have an LCD on the DL60, but it’s here in my office so I can tell every time it reboots by the fans), I figured something was wrong and hopped on the ILO to see what was going on where I was greeted by a BSOD loop – STOP 0x00000133 (DPC WATCHDOG VIOLATION).  After troubleshooting (I couldn’t even get it to boot into the current build of Microsft DaRT as it too would cause a BSOD), I decided to make a Windows Server 2016 installation USB key from the setup DVD using Rufus.  (I could have booted off the DVD ISO image via the ILO, but the ILO emulates USB 2 as opposed to the physical USB 3 ports in the server, so it would have been much slower).  I copied the boot.wim to my C: drive and injected the most recent driver pack into it from the current HPE Support Pack for Proliant (in the root of the HPSPP DVD, called \WIN_DRV) before copying the wim back to the USB key.

I then booted off the USB key and when the Windows Server 2016 Setup Window opened, I hit Shift + F10 to open a command prompt.  I deleted C:\Windows\winsxs\pending.xml then ran “wpeutil reboot“.  As soon as the F9 to F12 function keys became available in the Proliant POST screen, I hit F11, which eventually brought me to the Proliant boot menu.  Here I selected “Windows Boot Manager” and then immediately started hitting F8 to get to the Windows boot options.  This allowed me to select “Last Known Good Configuration”, which allowed the server to boot into Windows without a Stop 0x00000133 (note – selecting just “Last Known Good Configuration”, or deleting just C:\Windows\winsxs\pending.xml won’t help – I found you must do both).

I retried the Windows Update a couple of times only to have the same thing happen again and again, so I followed the above steps again and again to get Windows Server 2016 running again.  Then I Googled the KB number (KB4093119) and went to the Microsoft Support article about it.  At the bottom of the support article, under “How to get this update” is a link to the stand-alone installation package on the Microsoft Update Catalog website.  This allowed me to download the .msu for KB4093119, which I saved to C:\DL\UPDATES as KB4093119.msu.

I rebooted once again off the Windows Server 2016 installation USB key.  When the Windows Server 2016 Setup Window opened, I hit Shift + F10 to open a command prompt.  In the command prompt, I ran the following two commands:

md c:\temp
dism /image:c:\ /add-package /packagepath:c:\dl\updates\KB4093119.msu /scratchdir:c:\temp

Once dism finished successfully, I rebooted the server with “wpeutil reboot

Windows Server 2016 finally booted successfully with the Cumulative Update installed, and I did another check for updates – there were no updates left to install.  I suspect these same steps would work on a Windows 10 machine that is having similar issues – although I don’t know if “Last Known Good Configuration” is an option with the most current version of Windows 10.

As always – Use any tips, tricks, or scripts I post at your own risk.

HOWTO: Approve just the #WSUS updates that are needed using #PoshWSUS and #PowerShell on a schedule

Recently we deployed a new “do-all, be-all” server (AD, WSUS, etc) to a small client with very poor bandwidth availability.  Just allowing WSUS to auto-approve every published update across all categories and all select products just wasn’t an option (and really shouldn’t be anyways) because it would have taken a month to download them over 3Mbps DSL.  So I wanted to only approve and download those updates that were needed, as opposed to WSUS built-in process of automatic approve everything in a classification or product then immediately trying to download it.

For a bit more background information, typically we configure WSUS at our smaller “hands off clients” to check for new updates at 4 am daily (although I’m considering switching to manual synchronization and using PowerShell to invoke the check on a schedule that is more “robost” shall we say.  This gives us about a day to vet the basic updates on our own production system (which is typically close in configuration to our smaller “hands off clients”) before the updates auto-approve after synchronization at the client locations.

In the past I have played with the native Windows Server Update Services PowerShell cmdlets, but lets be honest, they are only about half baked and certainly not ready for enterprise IT production (I’m really not sure what Microsoft was thinking).  The inability to auto-accept EULAs is bad.  Even worse is that if there is a EULA waiting to be accepted, Approve-WsusUpdate just pukes, exits, and refuses to acknowledge, and refuses to approve any other updates (even without EULAs on them).  So this means there really is no hands free way provided by Microsoft to auto-approve just the needed updates as they come out…

After some research, I settled on PoshWSUS.  In my honest opinion however, it is not well documented or supported (but it is good given the resources the author likely has at his disposal)…  It took me 4 hours of scouring Google and trial and error to come up with this basic script, which simply looks at all the needed updates and approves them, in the fewest lines of PowerShell possible.  In the end I had to mash together both the native WSUS PowerShell cmdlets and the PoshWSUS cmdlets to make this work, but it does work (so far anyways).

So, download PoshWSUS_2_3_1_0.zip from GitHub, drop it in C:\Windows\System32\WindowsPowerShell\v1.0\Modules and lets go…

First of all, we need to find just updates that are needed – err – rather **ALL** the updates that are needed.  I didn’t find a way in PoshWSUS to do this, but the native WSUS cmdlets let you do this.  Great – but you can’t pipe these updates (directly out of Get-WsusUpdate) into PoshWSUS’s Approve-PSWSUSUpdate, and I couldn’t find any way to get all needed updates out of Get-PSWSUSUpdate (which can pipe to Approve-PSWSUSUpdate), so I needed to get creative…  Get-PSWSUSUpdate, using a piped Where will allow you to select by the UpdateId, and Get-WsusUpdate will output UpdateId…  Hmmm….

Now – how to get the UpdateId from Get-WsusUpdate to Get-PSWSUSUpdate…  How about by piping the UpdateId via >> to a text file, then use foreach to read that text file into Get-PSWSUSUpdate, which then pipes to Approve-PSWSUSUpdate…  It’s ugly, it’s slow, but hey it works, and should only take a minute or two to run even on the heaviest of Patch Tuesdays…  And along the way, the text file needs cleaned up before being fed into Get-PSWSUSUpdate.

So here is the entire process (save this script as C:\Windows\WSUS_Updates_Approval.ps1)…

Import-Module PoshWSUS
$DNSDomainName=$env:userdnsdomain
$ThisComputersFQDN = "$env:computername"+"."+"$DNSDomainName"
Connect-PSWSUSServer -WsusServer $ThisComputersFQDN -Port 8530
$UpdateIDFile = "C:\Windows\Temp\UpdateIDFile.txt"
If (Test-Path $UpdateIDFile){Remove-Item $UpdateIDFile}
Get-WsusUpdate -Approval Unapproved -Status Needed | Select-Object UpdateId >> $UpdateIDFile
(Get-Content $UpdateIDFile| Select-Object -Skip 3) | Set-Content $UpdateIDFile
(Get-Content $UpdateIDFile| Foreach {$_.TrimEnd()}) | Set-Content $UpdateIDFile
$Groups = Get-PSWSUSGroup -Name 'All Computers'
foreach ($NewUpdateID in get-content $UpdateIDFile) {Get-PSWSUSUpdate | Where {$_.UpdateID -eq "$NewUpdateID"} | Approve-PSWSUSUpdate -Action Install -Group $Groups -Confirm:$false}

Finally, we just need to schedule this script to run on a regular basis… I typically schedule it to run every 15 minutes.  To do that, run this in an Administrative Command Prompt:

schtasks /create /tn "WSUS Updates Approval" /tr "\"C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe\" -ExecutionPolicy RemoteSigned -noprofile -File C:\Windows\WSUS_Updates_Approval.ps1" /sc minute /mo 15 /st 00:02:00  /rp "*" /ru "%userdomain%\%username%"

As always – Use any tips, tricks, or scripts I post at your own risk.

HOWTO: Nuke Windows Server 2012R2 #WSUS and start fresh without actually having to uninstall, reinstall, or even reboot!

Recently I ended up in a situation where WSUS (Windows Server Update Services) running on a Windows Server 2012 R2 box was messed up and would have required more time to fix and clean up than I wanted to invest.  So I started looked into my notes for resetting WSUS to defaults to start over, but I found conflicting information on it in the notes I had made over the years.  So – off to my R&D lab for testing and here is the working results…

First, I always install SQL Express Advanced (with tools) as the default unnamed instance (MSSQLSERVER) on the server before I enable and configure the WSUS role (both of which I do via scripting).  Second, I always put the WSUS content folders in D:\SHARED\WSUS.  By doing so, I find this:

  • allows me more control over how the server is setup
  • allows me to cookie cutter my customers’ environments
  • allows my support team to already be knowledgeable with the setup before they even see / support it
  • allows me to script the setup, which saves me time!!!

So, back to my current issue – nuking WSUS and starting it fresh.  The general process to do so is:

  • Stop the WSUS, WWW, and BITS services
  • Delete the WSUS database from the MSSQL instance
  • Delete and recreate the WSUS content folders
  • Restart the WWW and WSUS services (BITS will start on its own when needed)
  • Run the WSUS post-install setup utility to provision WSUS to the MSSQL instance and set the content directory
  • Run the WSUS Setup Configuration Wizard

Of course, it’s very simple to script all that into a repeatable process for my support team in case they run into issues with a client’s WSUS, and because I script the initial setup of WSUS, all our clients WSUS servers are configured the same, which means my script will work on all my clients WSUS servers.

So to Nuke WSUS and start it fresh, open an Administrative Command prompt and paste in the following commands (note the -Q in sqlcmd is case sensitive).

net stop /y WsusService
net stop /y w3svc
net stop /y bits
sqlcmd -Q "drop database [SUSDB]"
rd /q /s D:\SHARED\WSUS
md D:\SHARED\WSUS
net start w3svc
net start WsusService
"C:\Program Files\Update Services\Tools\wsusutil.exe" postinstall SQL_INSTANCE_NAME="%computername%" CONTENT_DIR=D:\SHARED\WSUS
"C:\Program Files \Update Services\AdministrationSnapin\wsus.msc"

You should now have a nice clean WSUS server to start fresh with.

As always – Use any tips, tricks, or scripts I post at your own risk.

 

HOWTO: Install Windows Server 2012 R2 #WSUS via script (including all prereqs)

It’s probably no secret that I’m a fan of scripting.  I love the ability to be able to get consistent and even results on every job, install or upgrade my support team or I do.  The end result is it makes installations and updates faster.  And it definitely makes everyone’s life easier when trouble finds you – all you have to do is compare your broken system to a working system that is configured the same way and you can usually figure out the issue.  Also I’ve been around the industry long enough to remember DOS 5, DOS 6.22, and early versions of OS/2 (not to mention the Vic20, C64 and Amiga 500s I had over the years) – I’m generally always faster on a keyboard than with a mouse (although I actually use a trackball when I’m at my desk), albeit sometimes the mouse is just simpler – but those cases are few and far between in an enterprise class IT world.

One of the tools I end up often having to install from scratch at client locations is WSUS (Windows Server Update Services).  And it’s a pain in the ass to install.  You need to install IIS, you need to install SQL, you need to install WSUS, and then you have to configure WSUS itself.  All time consuming if you use Server Manager.  But hey – breakout the Command Prompt and PowerShell and you can be done in no time with a few simple commands.

Before you follow these instructions, make sure the server you are doing this on is an already updated Windows 2012 R2 Server with all current Microsoft Updates (otherwise it may cause you extra bullshit and grief later on).  Then after WSUS is installed, be to check for Microsoft Updates (hey – you could probably use your shiny new WSUS server to do that!) again to update the prerequisites you had to install before even installing WSUS.

Speaking of those prerequisites, of course WSUS on Windows Server 2012 R2 has a few prerequisites.  The first one is IIS – it must be installed with specific features. Being lazy, I usually install most all the features with my catch-all IIS installation script.  Open an Administrative Command Prompt and run:

DISM.EXE /enable-feature /online /featurename:IIS-ASP /featurename:IIS-ASPNET /featurename:IIS-ASPNET45 /featurename:IIS-ApplicationDevelopment /featurename:IIS-ApplicationInit  /featurename:IIS-BasicAuthentication  /featurename:IIS-CGI  /featurename:IIS-CertProvider /featurename:IIS-ClientCertificateMappingAuthentication /featurename:IIS-CommonHttpFeatures /featurename:IIS-CustomLogging  /featurename:IIS-DefaultDocument  /featurename:IIS-DigestAuthentication  /featurename:IIS-DirectoryBrowsing /featurename:IIS-HealthAndDiagnostics /featurename:IIS-HttpCompressionDynamic  /featurename:IIS-HttpCompressionStatic /featurename:IIS-HttpErrors /featurename:IIS-HttpLogging /featurename:IIS-HttpRedirect  /featurename:IIS-HttpTracing /featurename:IIS-IIS6ManagementCompatibility /featurename:IIS-IISCertificateMappingAuthentication  /featurename:IIS-IPSecurity  /featurename:IIS-ISAPIExtensions /featurename:IIS-ISAPIFilter /featurename:IIS-LegacyScripts /featurename:IIS-LegacySnapIn /featurename:IIS-LoggingLibraries  /featurename:IIS-ManagementConsole /featurename:IIS-ManagementScriptingTools  /featurename:IIS-ManagementService /featurename:IIS-Metabase  /featurename:IIS-NetFxExtensibility  /featurename:IIS-NetFxExtensibility45 /featurename:IIS-ODBCLogging /featurename:IIS-Performance /featurename:IIS-RequestFiltering /featurename:IIS-RequestMonitor /featurename:IIS-Security /featurename:IIS-ServerSideIncludes  /featurename:IIS-StaticContent /featurename:IIS-URLAuthorization /featurename:IIS-WMICompatibility /featurename:IIS-WebServer  /featurename:IIS-WebServerManagementTools /featurename:IIS-WebServerRole /featurename:IIS-WebSockets /featurename:IIS-WindowsAuthentication /featurename:NetFx4Extended-ASPNET45

Next, we need to install SQL Express.  I’ve been pretty much standardized on SQL Express 2014 SP2 with tools (SQLEXPRWT_x64_ENU.exe) for a while now.  From the command prompt you are going to extract it to C:\TEMP then run an unattended SQL setup using these two commands from an Administrative Command Prompt (**note:  this command line expects you to be using SQL Express 2014 SP2 and not a previous version of SQL Express, even if WSUS supports it**):

SQLEXPRWT_x64_ENU.exe /q /x:"c:\TEMP\SQLEXPRWT_x64_ENU"
c:\TEMP\SQLEXPRWT_x64_ENU\SETUP.exe /ACTION=Install /ADDCURRENTUSERASSQLADMIN /AGTSVCACCOUNT="NT AUTHORITY\LOCAL SERVICE" /AGTSVCSTARTUPTYPE="AUTOMATIC" /BROWSERSVCSTARTUPTYPE="AUTOMATIC" /FEATURES=SQLENGINE,Replication,Tools /IACCEPTSQLSERVERLICENSETERMS /INDICATEPROGRESS /INSTANCEID=MSSQLSERVER /INSTANCENAME=MSSQLSERVER /NPENABLED=1 /QS /ROLE=AllFeatures_WithDefaults /SQLSVCACCOUNT="NT AUTHORITY\SYSTEM" /SQLSYSADMINACCOUNTS="Administrators" /SQMREPORTING=0 /TCPENABLED=1

WSUS also has a prerequisite that IIS_WPG, NETWORK, NETWORK SERVICE, and SERVICE have “Log on a service” rights on the WSUS server.  Unfortunately, this part is a mouse job (I haven’t done the research yet to script this).  Start the Local Security Policy Management Console on the WSUS server. Navigate to Local Policies –> User Rights Assignment branch, edit the “Log on as a service” setting, and add the following four accounts (you should be able to cut and paste these into the add accounts dialog box):

IIS_WPG; NETWORK; NETWORK SERVICE; SERVICE

There is a chance IIS_WPG won’t exist, don’t worry about it, just remove it and keep on going (if it doesn’t exist now, then it isn’t going to exist when WSUS is installed either).  After closing Local Security Policy, don’t forget to update the policy with “gpupdate /force” from the Administrative Command Prompt before continuing.

Now open an Administrative PowerShell and run:

Install-WindowsFeature -Name UpdateServices-Services, UpdateServices-DB -IncludeManagementTools

Once Install-WindowsFeature completes, back in the Administrative Command Prompt run:

"C:\Program Files\Update Services\Tools\wsusutil.exe" postinstall SQL_INSTANCE_NAME="%computername%" CONTENT_DIR=D:\SHARED\WSUS

When wsusutil.exe has completed, if KB3148812, which is a May 2016 Windows Update (or later version of it, which is required for distributing Windows 10 Anniversary or new version updates) has not yet been installed on the server (this is why told you earlier to start with a fully patched Windows 2012 R2 Server because you don’t need to deal with this bullshit if you do), after installing KB3148812 (or whatever Microsoft superseded it with) you must open an Administrative PowerShell and run:

Install-WindowsFeature -Name NET-WCF-HTTP-Activation45

Then, still related to KB3148812 (see my bullshit comment above – you should have started with a fully patched Windows 2012 R2 Server to begin with), open an Administrative Command Prompt and run:

"C:\Program Files\Update Services\Tools\wsusutil.exe" postinstall /servicing

Finally, after taking care of all the KB3148812 bullshit (I told you that you should have started with a fully patched Windows 2012 R2 Server to begin with), or if you smart enough to actually take my advice and start with a fully updated Windows 2012 R2 server, it’s finally time to the launch the WSUS console and run the configuration wizard by running this from the Administrative Command Prompt :

"C:\Program Files\Update Services\AdministrationSnapin\wsus.msc"

Lastly, if you are lazy like me, you’ll just next, next, next all the way through the WSUS Setup Wizard, skipping the “Synchronize Now” option.  Once you end up at the normal WSUS console, open a new Administrative PowerShell and run:

Get-WsusServer | Get-WsusClassification | Where-Object -FilterScript {$_.Classification.Title -Eq "Applications"} | Set-WsusClassification Get-WsusServer | Get-WsusClassification | Where-Object -FilterScript {$_.Classification.Title -Eq "Critical Updates"} | Set-WsusClassification Get-WsusServer | Get-WsusClassification | Where-Object -FilterScript {$_.Classification.Title -Eq "Definition Updates"} | Set-WsusClassification Get-WsusServer | Get-WsusClassification | Where-Object -FilterScript {$_.Classification.Title -Eq "Security Updates"} | Set-WsusClassification Get-WsusServer | Get-WsusClassification | Where-Object -FilterScript {$_.Classification.Title -Eq "Service Packs"} | Set-WsusClassification Get-WsusServer | Get-WsusClassification | Where-Object -FilterScript {$_.Classification.Title -Eq "Tools"} | Set-WsusClassification Get-WsusServer | Get-WsusClassification | Where-Object -FilterScript {$_.Classification.Title -Eq "Update Rollups"} | Set-WsusClassification Get-WsusServer | Get-WsusClassification | Where-Object -FilterScript {$_.Classification.Title -Eq "Updates"} | Set-WsusClassification Get-WsusServer | Get-WsusClassification | Where-Object -FilterScript {$_.Classification.Title -Eq "Drivers"} | Set-WsusClassification -Disable Get-WsusServer | Get-WsusClassification | Where-Object -FilterScript {$_.Classification.Title -Eq "Feature Packs"} | Set-WsusClassification -Disable Get-WsusServer | Get-WsusClassification | Where-Object -FilterScript {$_.Classification.Title -Eq "Upgrades"} | Set-WsusClassification -Disable Get-WsusServer | Get-WsusProduct | Where-Object -FilterScript {$_.Product.Title -Eq "CAPICOM"} | Set-WsusProduct Get-WsusServer | Get-WsusProduct | Where-Object -FilterScript {$_.Product.Title -Eq "Developer Tools, Runtimes, and Redistributables"} | Set-WsusProduct Get-WsusServer | Get-WsusProduct | Where-Object -FilterScript {$_.Product.Title -Eq "Dictionary Updates for Microsoft IMEs"} | Set-WsusProduct Get-WsusServer | Get-WsusProduct | Where-Object -FilterScript {$_.Product.Title -Eq "Exchange Server 2007 and Above Anti-spam"} | Set-WsusProduct Get-WsusServer | Get-WsusProduct | Where-Object -FilterScript {$_.Product.Title -Eq "Exchange Server 2013"} | Set-WsusProduct Get-WsusServer | Get-WsusProduct | Where-Object -FilterScript {$_.Product.Title -Eq "Exchange Server 2016"} | Set-WsusProduct Get-WsusServer | Get-WsusProduct | Where-Object -FilterScript {$_.Product.Title -Eq "Expression"} | Set-WsusProduct Get-WsusServer | Get-WsusProduct | Where-Object -FilterScript {$_.Product.Title -Eq "Expression Design 1"} | Set-WsusProduct Get-WsusServer | Get-WsusProduct | Where-Object -FilterScript {$_.Product.Title -Eq "Expression Design 2"} | Set-WsusProduct Get-WsusServer | Get-WsusProduct | Where-Object -FilterScript {$_.Product.Title -Eq "Expression Design 3"} | Set-WsusProduct Get-WsusServer | Get-WsusProduct | Where-Object -FilterScript {$_.Product.Title -Eq "Expression Design 4"} | Set-WsusProduct Get-WsusServer | Get-WsusProduct | Where-Object -FilterScript {$_.Product.Title -Eq "Expression Media 2"} | Set-WsusProduct Get-WsusServer | Get-WsusProduct | Where-Object -FilterScript {$_.Product.Title -Eq "Expression Media V1"} | Set-WsusProduct Get-WsusServer | Get-WsusProduct | Where-Object -FilterScript {$_.Product.Title -Eq "Expression Web 3"} | Set-WsusProduct Get-WsusServer | Get-WsusProduct | Where-Object -FilterScript {$_.Product.Title -Eq "Expression Web 4"} | Set-WsusProduct Get-WsusServer | Get-WsusProduct | Where-Object -FilterScript {$_.Product.Title -Eq "Microsoft SQL Server 2012"} | Set-WsusProduct Get-WsusServer | Get-WsusProduct | Where-Object -FilterScript {$_.Product.Title -Eq "Microsoft SQL Server 2014"} | Set-WsusProduct Get-WsusServer | Get-WsusProduct | Where-Object -FilterScript {$_.Product.Title -Eq "Microsoft SQL Server 2016"} | Set-WsusProduct Get-WsusServer | Get-WsusProduct | Where-Object -FilterScript {$_.Product.Title -Eq "Microsoft SQL Server 2017"} | Set-WsusProduct Get-WsusServer | Get-WsusProduct | Where-Object -FilterScript {$_.Product.Title -Eq "Microsoft SQL Server Management Studio v17"} | Set-WsusProduct Get-WsusServer | Get-WsusProduct | Where-Object -FilterScript {$_.Product.Title -Eq "New Dictionaries for Microsoft IMEs"} | Set-WsusProduct Get-WsusServer | Get-WsusProduct | Where-Object -FilterScript {$_.Product.Title -Eq "Office 2002/XP"} | Set-WsusProduct Get-WsusServer | Get-WsusProduct | Where-Object -FilterScript {$_.Product.Title -Eq "Office 2003"} | Set-WsusProduct Get-WsusServer | Get-WsusProduct | Where-Object -FilterScript {$_.Product.Title -Eq "Office 2007"} | Set-WsusProduct Get-WsusServer | Get-WsusProduct | Where-Object -FilterScript {$_.Product.Title -Eq "Office 2010"} | Set-WsusProduct Get-WsusServer | Get-WsusProduct | Where-Object -FilterScript {$_.Product.Title -Eq "Office 2013"} | Set-WsusProduct Get-WsusServer | Get-WsusProduct | Where-Object -FilterScript {$_.Product.Title -Eq "Office 2016"} | Set-WsusProduct Get-WsusServer | Get-WsusProduct | Where-Object -FilterScript {$_.Product.Title -Eq "Report Viewer 2005"} | Set-WsusProduct Get-WsusServer | Get-WsusProduct | Where-Object -FilterScript {$_.Product.Title -Eq "Report Viewer 2008"} | Set-WsusProduct Get-WsusServer | Get-WsusProduct | Where-Object -FilterScript {$_.Product.Title -Eq "Report Viewer 2010"} | Set-WsusProduct Get-WsusServer | Get-WsusProduct | Where-Object -FilterScript {$_.Product.Title -Eq "SDK Components"} | Set-WsusProduct Get-WsusServer | Get-WsusProduct | Where-Object -FilterScript {$_.Product.Title -Eq "SQL Server"} | Set-WsusProduct Get-WsusServer | Get-WsusProduct | Where-Object -FilterScript {$_.Product.Title -Eq "SQL Server 2000"} | Set-WsusProduct Get-WsusServer | Get-WsusProduct | Where-Object -FilterScript {$_.Product.Title -Eq "SQL Server 2005"} | Set-WsusProduct Get-WsusServer | Get-WsusProduct | Where-Object -FilterScript {$_.Product.Title -Eq "SQL Server 2008"} | Set-WsusProduct Get-WsusServer | Get-WsusProduct | Where-Object -FilterScript {$_.Product.Title -Eq "SQL Server 2008 R2"} | Set-WsusProduct Get-WsusServer | Get-WsusProduct | Where-Object -FilterScript {$_.Product.Title -Eq "SQL Server 2012 Product Updates for Setup"} | Set-WsusProduct Get-WsusServer | Get-WsusProduct | Where-Object -FilterScript {$_.Product.Title -Eq "SQL Server 2014-2016 Product Updates for Setup"} | Set-WsusProduct Get-WsusServer | Get-WsusProduct | Where-Object -FilterScript {$_.Product.Title -Eq "SQL Server Feature Pack"} | Set-WsusProduct Get-WsusServer | Get-WsusProduct | Where-Object -FilterScript {$_.Product.Title -Eq "Silverlight"} | Set-WsusProduct Get-WsusServer | Get-WsusProduct | Where-Object -FilterScript {$_.Product.Title -Eq "Visual Studio 2005"} | Set-WsusProduct Get-WsusServer | Get-WsusProduct | Where-Object -FilterScript {$_.Product.Title -Eq "Visual Studio 2008"} | Set-WsusProduct Get-WsusServer | Get-WsusProduct | Where-Object -FilterScript {$_.Product.Title -Eq "Visual Studio 2010"} | Set-WsusProduct Get-WsusServer | Get-WsusProduct | Where-Object -FilterScript {$_.Product.Title -Eq "Visual Studio 2010 Tools for Office Runtime"} | Set-WsusProduct Get-WsusServer | Get-WsusProduct | Where-Object -FilterScript {$_.Product.Title -Eq "Visual Studio 2012"} | Set-WsusProduct Get-WsusServer | Get-WsusProduct | Where-Object -FilterScript {$_.Product.Title -Eq "Visual Studio 2013"} | Set-WsusProduct Get-WsusServer | Get-WsusProduct | Where-Object -FilterScript {$_.Product.Title -Eq "Windows 10 LTSB"} | Set-WsusProduct Get-WsusServer | Get-WsusProduct | Where-Object -FilterScript {$_.Product.Title -Eq "Windows 2000"} | Set-WsusProduct Get-WsusServer | Get-WsusProduct | Where-Object -FilterScript {$_.Product.Title -Eq "Windows 7"} | Set-WsusProduct Get-WsusServer | Get-WsusProduct | Where-Object -FilterScript {$_.Product.Title -Eq "Windows Server 2008 R2"} | Set-WsusProduct Get-WsusServer | Get-WsusProduct | Where-Object -FilterScript {$_.Product.Title -Eq "Windows Server 2012 R2"} | Set-WsusProduct Get-WsusServer | Get-WsusProduct | Where-Object -FilterScript {$_.Product.Title -Eq "Windows Server 2016"} | Set-WsusProduct

You can then go back to the WSUS console, select Options, and now Products and Classifications.  The above PowerShell commands will give you a great Product and Classification base to start your WSUS server configuration from without all the mouse clicks during the configuration wizard.

Congratulations – you now have a fully installed and configured WSUS server!

As always – Use any tips, tricks, or scripts I post at your own risk.

Workaround: When the #Windows10 Windows Hello setup UI won’t open…

Recently while traveling on the road for 3 weeks, my brand new notebook with Windows 10 x64 Enterprise Edition (Fall Creators Update) started blue screening at boot (safe mode wouldn’t even start).  I really didn’t have the ability to take the time to troubleshoot it too deeply and none of the standard Windows 10 repair functions worked.  In the end I used “Reset my PC” which seems to have solved the blue screen of death, but left me with no installed applications (which really sucked) although my user profile was mostly left intact.  I actually had to use “Reset my PC” 3 days in a row at one point so I could work, and then finally my notebook seemed to return to a stable working condition – until Saturday morning that is.

Well enough was enough – I can’t trust the Windows installation not to give me grief again in the future, and since I use Veeam Agent for nightly backups, I decided to start over by booting WinPE and running Diskpart then Clean on my two SSDs.  I reinstalled Windows 10 x64 Enterprise Edition (Fall Creators Update), along with all the zBook 14u G4 drivers, and proceeded to setup my notebook like I always would.  After several hours of installing and configuring software and restoring about 1TB of data from Friday night’s Veeam backup, I had two things left to do.  Configure my fingerprint reader and encrypt my drives with Symantec PGP Corporate Desktop.

So Windows Hello needs a PIN before you can add a fingerprint – annoying as hell, but necessary… I mean – I have a strong password to protect my account – why the hell should I have to add relatively weak PIN to enable my finger print (but I will digress on this rant and get to what’s important here).  So I go to Settings –> Account –> Sign-in Options and add my PIN.  Now that my PIN is added, I click on the Add Finger Print button…

2017.12.04 - 13.50.07 - SNAGIT - 0005

The Windows Hello setup UI opens and then immediately closes – basically just a flash…  I hit it again and still no go.  Uh-oh… Reboot?  Nope.  Drivers?  Nope.  After some Googling and removing all fingerprint data from the BIOS, I’m still no further ahead and out of leads.

Hmm – I used a new tool to customize my profile and did a few changes to my profile that I normally haven’t done in the past – what if it is my user profile fighting with UAC that is causing this?  So I log in as Administrator and find the “Finger Print” button is greyed out.  Ok, well maybe I need to do it as a standard user, so I open a command prompt and create a new user with:

net user testuser password /add

I logout as Administrator and login as testuser.  When I open Settings –> Account –> Sign-in Options, the Finger Print button is active, and when I click, it opens and allows me to scan my finger.  It appears the issue is my account / profile as opposed to something specifically in Windows 10.  This is a good news / bad news scenario.  I don’t need to reinstall Windows yet again, but I don’t want to have to spend hours re-configuring my profile yet again either.  So it’s time to get creative.

Here are the steps I used to get my finger print registered in Windows (note – this doesn’t fix the problem long term, it just works around it for now, which is all I need).

  1. Reboot the machine and login as Administrator
  2. Navigate to C:\Users and rename my user profile folder to C:\USERS\JBGEEK.good
  3. Open Regedit and navigate to HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\ProfileList and removed the entry that existed for my user profile.
  4. Open Computer Management, navigated to Users and Groups and removed my user account from the Administrators group.
  5. Logged out as Administrator
  6. Logged in with my userid (which creates a new user profile associated to my SID)
  7. Opened Settings –> Account –> Sign-in Options and clicked on Finger Print – the Windows Hello setup UI for finger prints opened and allowed me to register my fingers
  8. Rebooted Windows and verified I could login with my finger print from CTRL+ALT+DEL
  9. Rebooted Windows again to ensure my profile was unloaded and logged in as Administrator
  10. Navigated to C:\Users and renamed my newly create user profile to C:\USERS\JBGEEK.temp and then renamed my good user profile back to C:\USERS\JBGEEK
  11. Open Computer Management, navigated to Users and Groups and re-add my user account to the Administrators group.
  12. While in Computer Management, I also deleted my testuser and removed it’s user profile too.

I am now able to log into Windows using my finger print (without having lost my profile settings and data).  And although the Add Finger Print UI still doesn’t function for me, but I really don’t care because it is not like I plan to grow any new fingers anytime soon that I will need to register in Windows until the next reinstall!

Anyways – hopefully this blog will help someone else stuck in the same boat.

HOWTO: Install QLogic QConvergedConsole beside HPE 3Par SSMC and Veeam Backup & Recovery

Ok – I’ll admit it – I’m something of a vendor snob…  And my vendor of choice when it comes to Ethernet and fibre channel host connectivity is QLogic and HPE’s OEM products made by QLogic.  You just can’t beat the price or performance of the offerings, and the support that QLogic’s HPE OEM team gives you – they are second to none (a huge shout out to @ToddOwens_QLGC & Jim Burton – if you guys are reading this, thanks for all the amazing support over the years!).

One of the interesting things about QLogic is their branded applications generally work hand in hand with the OEM products they offer to various system manufacturers such as HPE, Dell, and Lenovo.  While I was attending a storage conference last week, I sat in on a presentation Jim and Todd were hosting.  During the presentation the talk turned to QLogic’s comprehensive adapter management tools, including the Web-based QCC (QConvergedConsole), which is supported on Windows, Linux, and Solaris.  QCC allows you to modify and configure your adapters (Ethernet, iSCSI, FCoE, and FC), upgrade the flash on them, perform FC ping and traceroute, and to view reports, statistics, and diagnostics of all the QLogic devices in your equipment – either locally or remote.

Given that QLogic devices are generally so bullet proof, and that the HPE Support Pack for Proliant takes care of my firmware updates, I rarely have a need to install and use QCC.  But today was a little different – I had a VMware host that suffered a Purple Screen of Death overnight, and while I was in the ILO power cycling it and looking for a reason for PSOD, I noticed that ILO was complaining that the 534FLR-SFP+ adapter was degraded because it was in FCoE mode and not connected (we don’t use FCoE).  Since I didn’t want to waste any more time playing around with the host before I brought it back online, I decided that I would load QCC on my management server at the site and see if I could disable FCoE mode remotely.

I never did find a way to disable the FCoE function via QCC – I only spent 3 minutes looking at it, so there may well be a way if I actually RTFM (that isn’t my style though), but this post isn’t about that.  This post is all about getting QCC to co-exist (temporarily anyways) on a server that already has HPE’s 3Par SSMC and / or Veeam Backup & Recovery installed on it.  QCC has been around a long time – longer that both SSMC and VBR, and as such has a few port conflicts that the guys at HPE and Veeam never took into consideration.  As a result, you can’t just fire up the QCC installer and expect it co-exist and run 7/24 right out of the box along side SSMC and VBR.

Once you have the QCC installer downloaded and extracted, there are a few things we need to do before firing up the installer.

First, lets check to make sure TCP ports 8080, 8443, and 111 are not in use.  We can accomplish this by opening an elevated command prompt and running:   netstat -ano | find “0.0.0.0:####”

2017-01-30-15-04-41-snagit-0067

In the example above, you can see that two of the three ports are in use.  Port 8443 is used by the application that has a PID of 38692, while port 111 is used by the application that has a PID of 30000.  Using Task Manager, or better yet my favorite tool for the job – Process Explorer, we can easily determine the applications that are hogging these ports if we enable the PID and Path columns and then sort of the PID.

2017-01-30-15-08-27-snagit-0068

So to get started, we need to stop (temporarily) SSMC and VBR’s vPower NFS service.

2017-01-30-13-56-34-snagit-0040

2017-01-30-13-57-00-snagit-0041

Now that we have stopped these two services, lets double check to make sure TCP ports 8080, 8443, and 111 are no longer in use.

2017-01-30-15-13-24-snagit-0069

So with all three ports now free and no longer in use, we can launch the QCC installer as Administrator (note – all screen snapshots are based on Windows 2012 R2 with QCC v5.4.0.41).  Click next a couple of times until you get to the “Please enter desired port number”.  This defaults to TCP 8080, which as we checked already above, is free to use, so go ahead and click Install.

2017-01-30-13-58-32-snagit-0046

Eventually the installer will prompt if you wish to restrict access to localhost.  No one else at my sites require access to QCC, so I’m ok with restricting access – I clicked yes (note it defaults to no, so if you just hit enter you answered no…)

2017-01-30-13-59-39-snagit-0048

Eventually, you’ll be prompted if you wish to enable security login.

2017-01-30-14-00-02-snagit-0050

Since this application is only going to be enabled temporarily when I actually need it on the management server on the management VLAN, and because I am restricting access to the localhost only, I left the checkbox cleared.  That said, you may wish enable security – and if you do, make sure you make a note of the credentials you set!  The default login id credentials if you didn’t change them is “QCC” with a password of “config”.  Click Next to continue.

Now you are prompted if you wish to enable SSL.  That is likely a good idea, even if you are restricting it to the localhost – so click yes.  This will automatically set the Tomcat7 engine to use TCP 8443 and you can not change this from the installer.

2017-01-30-14-00-21-snagit-0051

Finally you will be presented with the Done button.

2017-01-30-14-02-22-snagit-0052

Take note of the URL as you will need it shortly…  https://localhost:8443/QConvergeConsole/ or http://localhost:8080/QConvergeConsole/

Now we can go ahead and install the necessary management agents.  In my case I am going to install all of the management agents.

2017-01-30-14-02-55-snagit-0054

After we click Next, you’ll notice that the installer is installing the ONCPortmap service.  This runs on TCP 111.  If TCP 111 is already in use, the installer will hang, and hang, and hang…  This is why we stopped the Veeam vPower NFS service earlier.

2017-01-30-14-03-14-snagit-0055

Eventually the management agent will complete the install process.  When we install the next management agent, you’ll notice a warning about the ONCPortmap service – this is good!  It means the ONCPortmap installed and started successfully.

2017-01-30-14-04-05-snagit-0058

After we have all the management agents installed that we want or require, we can go back in the command prompt and check our port status again.

2017-01-30-14-08-10-snagit-0059

Now you can see that all three ports are in use – which means QCC is likely ready to go.  Sort of…  As I mentioned previously, because the ports used in QCC conflict with SSMC and Veeam vPower NFS, we can’t just leave things alone and expect all three apps to work in the future after a reboot.  In my environment SSMC and Veeam are more important than QCC, and I always want them to be started after a server reboot.  So we need to set the follow services to be manual start instead of automatic (which they are by default) so they don’t prevent SSMC or Veeam from starting.

  • ONC/RPC Portmapper
  • QLManagementAgentJava
  • QLogic Management Suite FastLinQ
  • QLogic Management Suite Java iQAgent
  • QLogicManagementSuitenQLRemote
  • Tomcat7

Once we have changed the startup type of these services to manual, then lets login using the URL we were shown above.

2017-01-30-14-08-58-snagit-0060

Now – in the Host Selection dialog box, type in localhost and hit the connect button.  You should be able to safely ignore any errors you may see.

2017-01-30-14-09-18-snagit-0061

Finally – the console is opened!  Lets make a simple cosmetic change to see if it works (so something that does not affects the performance or anything of the adapters).  Highlight one of the ports of one of your adapters (in my example below, Port 0 of the HP 533FLR-T) and click on the MBA Boot Cfg tab in the right hand pane.  In the Hide Setup Prompt drop-down box, pick the opposite of whatever is there (it is probably already disabled, so select enabled), then click the Apply button.

2017-01-30-14-11-55-snagit-0063

You’ll be prompted for a password.  This password, assuming you made no changes to the default setup will be “config”.  If you aren’t sure if this is correct, clear the checkbox that says save password.  If you leave it checked, and the password you put in is wrong, then you will need to log out of QCC and back in to be able to try a different password.

2017-01-30-14-12-14-snagit-0064

If you had the correct password, you’ll see a green banner advising you of a successful update!

2017-01-30-14-12-31-snagit-0065

Now all that is left is to make the changes you actually set out to do!  Of course, once you are finished, you have two choices – reboot the server to apply the changes and have SSMC and VBR startup on reboot, or ignore the reboot, manually stop all the QCC services (see the list above) and manually start the SSMC and VBR services.

Now that you have QCC installed, if you need to access it in the future, you can just stop SSMC and VBR, then start the necessary QCC services.  While it isn’t a perfect solution, it will allow QCC to coexist along side both SSMC and Veeam’s vPower NFS service.

As always – Use any tips, tricks, or scripts I post at your own risk.

HOWTO: Using #PowerShell to ensure a #Veeam USB Repository always has the correct drive letter

Some time ago I had a customer who switched completely to Veeam from Backup Exec (yeah baby!!!).  Offsite replication of any sort was out of the question as the customer simply couldn’t get the necessary bandwidth from any of the ISPs that serviced the area at anything that even approached an unreasonable price.  Instead we opted for a HPE StoreOnce for onsite and some sort of removable drive system for daily offsite.  The customer insisted on using Western Digital My Books (USB3) due to their capacity (6TB) and semi-ruggedness for the offline repo.  The drives are rotated out each day by a staff member for a new drive and they send the previous night’s drive offsite.

The My Books were originally setup using F: as the drive letter.  This would work great until someone plugged a USB key into the server and it was auto-assigned F: by Windows 2012R2, or if someone logged in who had a drive mapping of F:.  And then other days, Windows would randomly assign a drive letter other than F: even though F: was available.  At this point, Veeam would puke because it couldn’t find the repo anymore.  And occasionally the customer would forget to swap the drive for a fresh one (or even correct drive some days), which meant there wouldn’t be enough free space on the repo to complete that night’s backup (about 4.9TB is processed daily).

To work around all this, I ended up writing a PowerShell script to always map the My Book to Z:, clean the previous backups off of it (if any were found, or if the wrong drive with a different backup set was plugged in), and start the actual backup job.  And if the script can’t find the My Book or free disk capacity is less than 95% after attempting cleanup, it will abort the backup and send an email warning to the support team that the backup was aborted.  Rather than have Veeam schedule the job to run, I use Windows Task Scheduler instead to run the script, then use the Veeam PowerShell module to start the backup job from inside the script after I’ve verified Z: is present.

This script assumes that every rotated My Book drive has been formatted and has a volume label containing “WD MY BOOK”.  It also assumes the SMTP server is called mail dot whatever the DNS domain name of the machine running the script is (i.e. my AD is jbgeek.net, so mail.jbgeek.net).  All of my client sites have a cname for their Exchange server called “mail” created in their AD DNS zone – which means I can cut and paste the same script without modification from one client site to another, which cuts down the chance of an editing error, and makes it quicker to deploy new scripts when necessary for my team.

You will need to adjust the “Get-ChildItem -Path” (lines 21 to 26) directories to fit your environment, along with the $SendEmailTo variable (line 7) and the “Start-VBRJob -Job” job name (line 41).  Line 28 defines the cutoff point for disk capacity – if the available disk space is less than 95% of the drive capacity, the backup aborts.  Line 1 is commented it out – I generally always start my scripts with # start notepad++ script.filename so that I can just cut and paste it into a command prompt to create my script file – that way as I on-board each new customer, their sites are configured the same as existing sites.  Again, it cuts down the chances of an error and makes it quicker for my guys to deploy new scripts.

**NOTE – the following deletes data from your backup cartridges. Use any tips, tricks, or scripts I post at your own risk.  I accept zero liability and responsibility if you use these scripts!!!**

Here is the command line to create the scheduled task to run as the logged in user at 10pm each weekday (it will prompt you for your password when you run it in a command prompt):

schtasks /create /tn "Weekday Veeam USB Drive Backup" /tr "\"C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe\" -ExecutionPolicy RemoteSigned -noprofile -File  C:\Windows\Run_External_Drive_Veeam_Backup.ps1" /sc daily /st 22:00:00  /rp "*" /ru "%userdomain%\%username%"

Here is the Powershell script (updated 2017.01.17 to fix an error):

# start notepad++ C:\Windows\Run_External_Drive_Veeam_Backup.ps1

add-pssnapin Veeam*

$DnsDomain = Get-WmiObject -Class Win32_NTDomain -Filter "DSDirectoryServiceFlag='True'" | Select -ExpandProperty DnsForestName
$ThisComputerName = Get-WmiObject -Class Win32_ComputerSystem | Select -ExpandProperty Name
$SupportEmailAddress = "support@$DnsDomain"

# Get-WmiObject Win32_Volume | Where-Object {$_.DriveLetter -eq $null}
$drives = Get-WmiObject -Class win32_volume | Where-Object {$_.Label -match "WD MY BOOK"}
$i = 0
Foreach($drive in $drives)
{
#Set letter
$DriveLetter = "Z:"
Set-WmiInstance -input $drive -Arguments @{DriveLetter="$DriveLetter"}
$i++
}

if (Test-Path -path Z:\){
Get-ChildItem -Path "Z:\Veeam\Backups\All VMs - Z Drive" -Include *.vbm -recurse | foreach { $_.Delete()}
Get-ChildItem -Path "Z:\Veeam\Backups\All VMs - Z Drive" -Include *.vib -recurse | foreach { $_.Delete()}
Get-ChildItem -Path "Z:\Veeam\Backups\All VMs - Z Drive" -Include *.vbk -recurse | foreach { $_.Delete()}
Get-ChildItem -Path "Z:\Veeam\Backups\All VMs" -Include *.vbm -recurse | foreach { $_.Delete()}
Get-ChildItem -Path "Z:\Veeam\Backups\All VMs" -Include *.vib -recurse | foreach { $_.Delete()}
Get-ChildItem -Path "Z:\Veeam\Backups\All VMs" -Include *.vbk -recurse | foreach { $_.Delete()}
$externaldrivelabel = Get-WmiObject -Class win32_volume | Where-Object {$_.Label -match "WD MY BOOK"} | Select -ExpandProperty label
Get-WmiObject -Class win32_volume | Where-Object {$_.Label -match "WD MY BOOK"} | % { if (($_.FreeSpace/$_.Capacity) -le '0.95' )
{Send-MailMessage -From "$($ThisComputerName.ToUpper())@$($DnsDomain.ToUpper())" -To "$SupportEmailAddress" `
-Subject "Backup Aborted due to low space on $($externaldrivelabel)." `
-Body  "Aborting nightly Veeam backup to $($externaldrivelabel) (Z:) on $($ThisComputerName.ToUpper()).$($DnsDomain.ToUpper()) due to low disk space." `
-Priority High -DNO onSuccess, onFailure -SmtpServer "mail.$DnsDomain"
break}
}
}
if (Test-Path -path Z:\){
Send-MailMessage -From "$($ThisComputerName.ToUpper())@$($DnsDomain.ToUpper())" -To "$SupportEmailAddress" `
-Subject "Now starting nightly Veeam backup to $($externaldrivelabel)." `
-Body  "Now starting nightly Veeam backup to $($externaldrivelabel) (Z:) on $($ThisComputerName.ToUpper()).$($DnsDomain.ToUpper())." `
-Priority High -DNO onSuccess, onFailure -SmtpServer "mail.$DnsDomain"
Start-VBRJob -Job "All VMs - Z Drive" -FullBackup -RunAsync
break
}else{
Send-MailMessage -From "$($ThisComputerName.ToUpper())@$($DnsDomain.ToUpper())" -To "$SupportEmailAddress" `
-Subject "Error:  No external backup drive (Z:) found on $($ThisComputerName.ToUpper()).$($DnsDomain.ToUpper())!!!" `
-Body  "Error:  Cannot find or mount the external backup drive (Z:) on $($ThisComputerName.ToUpper()).$($DnsDomain.ToUpper())!!!  Now aborting nightly backup to external hard drive!!!" `
-Priority High -DNO onSuccess, onFailure -SmtpServer "mail.$DnsDomain"
break
}

As always – Use any tips, tricks, or scripts I post at your own risk.

HOWTO: Converting from BackupExec to #Veeam when using RDX drives

Ok – I’m completely done with Backup Exec when it comes to VMware.  I’ve been selling, supporting, certified on and even using Backup Exec for our own internal backups since it was Conner Backup Exec for Windows NT 3.1, way back in 1993.  Once upon a time, it was a great product – in fact it was the only product for backups that worked worth a damn.  But it’s reliability has dropped to nothing over the past 6 or 7 years.  Technical support has been off-shored and 99.9% of the time, if I am lucky enough to finally reach someone in technical support on the phone, I can’t understand a damn word they say due to their thick accent and shitty VOIP lines crossing the Pacific Ocean.  Today was the last straw with Backup Exec, their crappy bugs, and unreliable VMware backups.  So now it’s time to fully embrace the move to Veeam, which I’ve been considering for some time (note of disclosure – I am also a certified Veeam VMCE – v7, v8, & v9)

Several of my clients have single standalone ESXi hosts, an HPE StoreOnce appliance, a physical Windows Server 2012R2 with a RDX drive or two (for offline backups), and both Backup Exec and Veeam loaded on that Windows server.  Oh – and many, many, many RDX cartridges that have months of rotated backups on them that are all three quarters full.  I can’t just erase all these cartridges in one swoop and use them for Veeam backups.  And I certainly don’t want to have to log into the clients’ servers everyday to manually delete the old Backup Exec folders off the RDX (as they come up in rotation) so that there is enough room for the nightly Veeam backup.  And finally, even though I’m dumping Backup Exec for my VMware backups, I still need to use Backup Exec to backup the 2012R2 physical instance to the same RDX cartridge that Veeam is going to use (atleast until Veeam releases their next project).  So what do I do?

A little PowerShell scripting to the rescue – that is what I am going do!

After going through a sampling of several RDX cartridges at several different client sites, I’ve determined that when Backup Exec runs with GRT enabled it dumps those backed up VMs in IMGxxxxxx folders on the root of the RDX drive (including the VMDKs).  I also discovered (or at least in the environments that I’ve setup) that GRT enabled application backups (not VMs, but rather SQL, AD, Exchange) will also be in an IMG folder with either a file called ntds.dit or edb.chk, and sometimes both!  In my case, my 2012R2 server has SQL and AD on it, so I want to be careful not to delete IMG folders that potentially contain my SQL and AD backups (which could screw Backup Exec up even more than normal when it uses that cartridge again for the 2012R2 server).

In the end, I setup the RDX drive as a new rotated drive repository in Veeam (prior to this Veeam only backed up to the HPE StoreOnce).  I then create a new Veeam job that did active fulls to the RDX drive every night (with a restore points to keep of 1).  In the job’s Advanced Settings menu, I added a pre-run script that runs C:\Windows\Remove_BackupExec_IMG_Folders.cmd.  This script in turn launches a PowerShell script that deletes all the IMGxxxxxx folders off the RDX drive except IMG folders that contain either ntds.dit or edb.chk.

**NOTE – the following deletes data from your backup cartridges. Use any tips, tricks, or scripts I post at your own risk.  I accept zero liability and responsibility if you use these scripts!!!**

Here is the contents of my batch file.

rem start notepad++ "C:\Windows\Remove_BackupExec_IMG_Folders.cmd"
PowerShell -NoProfile -ExecutionPolicy Bypass -Command "& {Start-Process PowerShell -ArgumentList '-NoProfile -ExecutionPolicy Bypass -File ""C:\Windows\Remove_BackupExec_IMG_Folders.ps1""' -Verb RunAs}"
exit/b

Here is the contents of the PowerShell script to remove the IMGxxxxxx folders (adjust the drive letter accordingly)

# start notepad++ "C:\Windows\Remove_BackupExec_IMG_Folders.ps1"
foreach ($i in Get-ChildItem R:\IMG*)
{if ((test-path "$i\ntds.dit") -eq $False -and (test-path "$i\edb.chk") -eq $False) {Remove-Item $i -force -recurse -confirm:$false}}

But wait! There is more!

Because I am still going to have to suffer with Backup Exec a while longer to backup my 2012R2 server, I need to make sure my nightly Backup Exec job doesn’t eject the RDX cartridge on me before Veeam finishes it’s RDX job.  To ensure this, I disabled the scheduled RDX jobs on my Backup Exec server.  Fortunately, Backup Exec includes a PowerShell module called BEMCLI.  So I wrote a second set of scripts as it was simply a matter of starting PowerShell from a script, importing the module, and starting the job.  So this time my scripts are a post-job script to start the Backup Exec job only after the Veeam job completes.

2017-01-11-16-33-43-snagit-0002

Here is the batch file to launch PowerShell.

rem start notepad++ "C:\Windows\Start_BE_UTIL01_RDX_JOB.cmd"
PowerShell -NoProfile -ExecutionPolicy Bypass -Command "& {Start-Process PowerShell -ArgumentList '-NoProfile -ExecutionPolicy Bypass -File ""C:\Windows\Start_BE_UTIL01_RDX_JOB.ps1""' -Verb RunAs}"
exit/b

And here is the PowerShell script to start the Backup Exec job called “23:10 UTIL01 RDX-Full”.

# start notepad++ "C:\Windows\Start_BE_UTIL01_RDX_JOB.ps1"
Import-Module BEMCLI
Get-BEJob -Name "23:10 UTIL01 RDX-Full" | Start-BEJob -confirm:$false

Now when my Veeam backup job to RDX starts, it deletes all the IMGxxxxxx folders off the RDX drive (unless those folders contain either ntds.dit or edb.chk), and when it completes, it starts the remaining Backup Exec job, which ultimately ejects the RDX cartridge when it completes.

As always – Use any tips, tricks, or scripts I post at your own risk.

HOWTO: Silently remove old VMware vCenter 5.x apps/tools and install the newest 6.x ones

It appears that VMware has finally figured out how to make vSphere 6 stable, which means it is finally time for my team to start migrating our clients off vSphere 5.5 and onto vSphere 6.    Upgrading a vSphere host takes all of 60 seconds with esxcli followed by a reboot of the host.  Upgrading all the apps and tools to manage the vSphere hosts however can take hours if doing it manually across all the machines in a domain though.

Like many of you (I’m sure), we generally have the various VMware apps and tools such as VIClient, PowerCLI, VMRC, Client Integration Plugin, and Update Manager client installed on multiple machines throughout the client’s computer system.  It’s time consuming and a real pain in the butt to go into Add/Remove programs and manually uninstall all the old 5.x tools and then manually install all the new 6.0 tools on each of these machines.  So, after a bit of testing and troubleshooting, I’ve come up with a series of one liners to cut and paste into an administrative command prompt to do all the time consuming pain the butt work without actually doing any of the work myself… Using these scripts, I can generally remove all the old 4.x and 5.x software and install all the new 6.0 software in less than 5 minutes per machine.

And I use our inventory and software management system to determine which machines have 5.x apps installed on them before I ever begin so I can target just the machines I need to without wasting time.  So basically once I have all ESXi hosts upgraded to version 6, I use Remote Desktop Connection Manager to connect to each machine I have identified as having 5.x apps, open an Administrator command prompt and cut and paste all my command lines in (both uninstall and install).  Once that machine is cranking away, I move onto the next machine and start the same process over again, and then on to the next machine.  Generally the first machine is completed before I get the last machine even started.  Then it is just a matter of verifying the apps work as expected…

So first we want remove all existing VMware apps on the target machine except VMware Tools and VMware Update Manager (server, not client).  You should be able to cut and paste all 7 of these command lines into the administrative command prompt at the same time and they will run one after the other, silently uninstalling any installed application on the machine with a name that matches the search parameters. The 7th line will open Add/Remove programs for you so you can manually verify everything has been removed before continuing.

Important – make sure there are no opened/running browsers on the machine and that none of the VMware apps are opened (very important if you are doing this on a Windows server that allows Remote Desktop for Administration and another admin is logged into it at the same time!!!)

**Note – if you are still using a Windows based vCenter server – it likely wouldn’t be too wise to run these uninstall commands on the vCenter server – consider yourself warned**

start /wait wmic product where "name like 'vmware c%%'" call uninstall
start /wait wmic product where "name like 'vmware r%%'" call uninstall
start /wait wmic product where "name like 'vmware vix%%'" call uninstall
start /wait wmic product where "name like 'vmware vsphere c%%'" call uninstall
start /wait wmic product where "name like 'vmware vsphere p%%'" call uninstall
start /wait wmic product where "name like 'vmware vsphere update manager c%%'" call uninstall
start appwiz.cpl

So now we all our old version 5.x tools and apps removed from our management stations, so we can go ahead now and silently deploy our new version 6.x apps.  Again, you should be able to paste these 6 lines all at once into a command prompt and they will run sequentially and install the VIClient, VMware Remote Console, vSphere CLI, vSphere PowerCLI, and the Update Manager client.

 

start /wait \\SERVER\SETUP\VMWARE\ESXi60u02\VMware-viclient.exe /q /s /w /L1033 /v" /qr"
start /wait msiexec /qb- /i \\SERVER\SETUP\VMWARE\ESXi60u02\VMware-VMRC-9.0.0-4288332.msi EULAS_AGREED=1 AUTOSOFTWAREUPDATE=0 DATACOLLECTION=0
start /wait \\SERVER\SETUP\VMWARE\ESXi60u02\VMware-vSphere-CLI-6.0.0-3561779.exe  /s /v/qn
start /wait \\SERVER\SETUP\VMWARE\ESXi60u02\VMware-PowerCLI-6.3.0-3737840.exe  /s /v/qn 
start /wait \\SERVER\SETUP\VMWARE\ESXi60u02\VMware-UMClient.exe  /s /v/qn 
start appwiz.cpl

 

Add/Remove Programs should once again automatically open for you to manually verify that everything has installed correctly.

As always – Use any tips, tricks, or scripts I post at your own risk.