HOWTO: Install QLogic QConvergedConsole beside HPE 3Par SSMC and Veeam Backup & Recovery

Ok – I’ll admit it – I’m something of a vendor snob…  And my vendor of choice when it comes to Ethernet and fibre channel host connectivity is QLogic and HPE’s OEM products made by QLogic.  You just can’t beat the price or performance of the offerings, and the support that QLogic’s HPE OEM team gives you – they are second to none (a huge shout out to @ToddOwens_QLGC & Jim Burton – if you guys are reading this, thanks for all the amazing support over the years!).

One of the interesting things about QLogic is their branded applications generally work hand in hand with the OEM products they offer to various system manufacturers such as HPE, Dell, and Lenovo.  While I was attending a storage conference last week, I sat in on a presentation Jim and Todd were hosting.  During the presentation the talk turned to QLogic’s comprehensive adapter management tools, including the Web-based QCC (QConvergedConsole), which is supported on Windows, Linux, and Solaris.  QCC allows you to modify and configure your adapters (Ethernet, iSCSI, FCoE, and FC), upgrade the flash on them, perform FC ping and traceroute, and to view reports, statistics, and diagnostics of all the QLogic devices in your equipment – either locally or remote.

Given that QLogic devices are generally so bullet proof, and that the HPE Support Pack for Proliant takes care of my firmware updates, I rarely have a need to install and use QCC.  But today was a little different – I had a VMware host that suffered a Purple Screen of Death overnight, and while I was in the ILO power cycling it and looking for a reason for PSOD, I noticed that ILO was complaining that the 534FLR-SFP+ adapter was degraded because it was in FCoE mode and not connected (we don’t use FCoE).  Since I didn’t want to waste any more time playing around with the host before I brought it back online, I decided that I would load QCC on my management server at the site and see if I could disable FCoE mode remotely.

I never did find a way to disable the FCoE function via QCC – I only spent 3 minutes looking at it, so there may well be a way if I actually RTFM (that isn’t my style though), but this post isn’t about that.  This post is all about getting QCC to co-exist (temporarily anyways) on a server that already has HPE’s 3Par SSMC and / or Veeam Backup & Recovery installed on it.  QCC has been around a long time – longer that both SSMC and VBR, and as such has a few port conflicts that the guys at HPE and Veeam never took into consideration.  As a result, you can’t just fire up the QCC installer and expect it co-exist and run 7/24 right out of the box along side SSMC and VBR.

Once you have the QCC installer downloaded and extracted, there are a few things we need to do before firing up the installer.

First, lets check to make sure TCP ports 8080, 8443, and 111 are not in use.  We can accomplish this by opening an elevated command prompt and running:   netstat -ano | find “0.0.0.0:####”

2017-01-30-15-04-41-snagit-0067

In the example above, you can see that two of the three ports are in use.  Port 8443 is used by the application that has a PID of 38692, while port 111 is used by the application that has a PID of 30000.  Using Task Manager, or better yet my favorite tool for the job – Process Explorer, we can easily determine the applications that are hogging these ports if we enable the PID and Path columns and then sort of the PID.

2017-01-30-15-08-27-snagit-0068

So to get started, we need to stop (temporarily) SSMC and VBR’s vPower NFS service.

2017-01-30-13-56-34-snagit-0040

2017-01-30-13-57-00-snagit-0041

Now that we have stopped these two services, lets double check to make sure TCP ports 8080, 8443, and 111 are no longer in use.

2017-01-30-15-13-24-snagit-0069

So with all three ports now free and no longer in use, we can launch the QCC installer as Administrator (note – all screen snapshots are based on Windows 2012 R2 with QCC v5.4.0.41).  Click next a couple of times until you get to the “Please enter desired port number”.  This defaults to TCP 8080, which as we checked already above, is free to use, so go ahead and click Install.

2017-01-30-13-58-32-snagit-0046

Eventually the installer will prompt if you wish to restrict access to localhost.  No one else at my sites require access to QCC, so I’m ok with restricting access – I clicked yes (note it defaults to no, so if you just hit enter you answered no…)

2017-01-30-13-59-39-snagit-0048

Eventually, you’ll be prompted if you wish to enable security login.

2017-01-30-14-00-02-snagit-0050

Since this application is only going to be enabled temporarily when I actually need it on the management server on the management VLAN, and because I am restricting access to the localhost only, I left the checkbox cleared.  That said, you may wish enable security – and if you do, make sure you make a note of the credentials you set!  The default login id credentials if you didn’t change them is “QCC” with a password of “config”.  Click Next to continue.

Now you are prompted if you wish to enable SSL.  That is likely a good idea, even if you are restricting it to the localhost – so click yes.  This will automatically set the Tomcat7 engine to use TCP 8443 and you can not change this from the installer.

2017-01-30-14-00-21-snagit-0051

Finally you will be presented with the Done button.

2017-01-30-14-02-22-snagit-0052

Take note of the URL as you will need it shortly…  https://localhost:8443/QConvergeConsole/ or http://localhost:8080/QConvergeConsole/

Now we can go ahead and install the necessary management agents.  In my case I am going to install all of the management agents.

2017-01-30-14-02-55-snagit-0054

After we click Next, you’ll notice that the installer is installing the ONCPortmap service.  This runs on TCP 111.  If TCP 111 is already in use, the installer will hang, and hang, and hang…  This is why we stopped the Veeam vPower NFS service earlier.

2017-01-30-14-03-14-snagit-0055

Eventually the management agent will complete the install process.  When we install the next management agent, you’ll notice a warning about the ONCPortmap service – this is good!  It means the ONCPortmap installed and started successfully.

2017-01-30-14-04-05-snagit-0058

After we have all the management agents installed that we want or require, we can go back in the command prompt and check our port status again.

2017-01-30-14-08-10-snagit-0059

Now you can see that all three ports are in use – which means QCC is likely ready to go.  Sort of…  As I mentioned previously, because the ports used in QCC conflict with SSMC and Veeam vPower NFS, we can’t just leave things alone and expect all three apps to work in the future after a reboot.  In my environment SSMC and Veeam are more important than QCC, and I always want them to be started after a server reboot.  So we need to set the follow services to be manual start instead of automatic (which they are by default) so they don’t prevent SSMC or Veeam from starting.

  • ONC/RPC Portmapper
  • QLManagementAgentJava
  • QLogic Management Suite FastLinQ
  • QLogic Management Suite Java iQAgent
  • QLogicManagementSuitenQLRemote
  • Tomcat7

Once we have changed the startup type of these services to manual, then lets login using the URL we were shown above.

2017-01-30-14-08-58-snagit-0060

Now – in the Host Selection dialog box, type in localhost and hit the connect button.  You should be able to safely ignore any errors you may see.

2017-01-30-14-09-18-snagit-0061

Finally – the console is opened!  Lets make a simple cosmetic change to see if it works (so something that does not affects the performance or anything of the adapters).  Highlight one of the ports of one of your adapters (in my example below, Port 0 of the HP 533FLR-T) and click on the MBA Boot Cfg tab in the right hand pane.  In the Hide Setup Prompt drop-down box, pick the opposite of whatever is there (it is probably already disabled, so select enabled), then click the Apply button.

2017-01-30-14-11-55-snagit-0063

You’ll be prompted for a password.  This password, assuming you made no changes to the default setup will be “config”.  If you aren’t sure if this is correct, clear the checkbox that says save password.  If you leave it checked, and the password you put in is wrong, then you will need to log out of QCC and back in to be able to try a different password.

2017-01-30-14-12-14-snagit-0064

If you had the correct password, you’ll see a green banner advising you of a successful update!

2017-01-30-14-12-31-snagit-0065

Now all that is left is to make the changes you actually set out to do!  Of course, once you are finished, you have two choices – reboot the server to apply the changes and have SSMC and VBR startup on reboot, or ignore the reboot, manually stop all the QCC services (see the list above) and manually start the SSMC and VBR services.

Now that you have QCC installed, if you need to access it in the future, you can just stop SSMC and VBR, then start the necessary QCC services.  While it isn’t a perfect solution, it will allow QCC to coexist along side both SSMC and Veeam’s vPower NFS service.

As always – Use any tips, tricks, or scripts I post at your own risk.

HOWTO: Using #PowerShell to ensure a #Veeam USB Repository always has the correct drive letter

Some time ago I had a customer who switched completely to Veeam from Backup Exec (yeah baby!!!).  Offsite replication of any sort was out of the question as the customer simply couldn’t get the necessary bandwidth from any of the ISPs that serviced the area at anything that even approached an unreasonable price.  Instead we opted for a HPE StoreOnce for onsite and some sort of removable drive system for daily offsite.  The customer insisted on using Western Digital My Books (USB3) due to their capacity (6TB) and semi-ruggedness for the offline repo.  The drives are rotated out each day by a staff member for a new drive and they send the previous night’s drive offsite.

The My Books were originally setup using F: as the drive letter.  This would work great until someone plugged a USB key into the server and it was auto-assigned F: by Windows 2012R2, or if someone logged in who had a drive mapping of F:.  And then other days, Windows would randomly assign a drive letter other than F: even though F: was available.  At this point, Veeam would puke because it couldn’t find the repo anymore.  And occasionally the customer would forget to swap the drive for a fresh one (or even correct drive some days), which meant there wouldn’t be enough free space on the repo to complete that night’s backup (about 4.9TB is processed daily).

To work around all this, I ended up writing a PowerShell script to always map the My Book to Z:, clean the previous backups off of it (if any were found, or if the wrong drive with a different backup set was plugged in), and start the actual backup job.  And if the script can’t find the My Book or free disk capacity is less than 95% after attempting cleanup, it will abort the backup and send an email warning to the support team that the backup was aborted.  Rather than have Veeam schedule the job to run, I use Windows Task Scheduler instead to run the script, then use the Veeam PowerShell module to start the backup job from inside the script after I’ve verified Z: is present.

This script assumes that every rotated My Book drive has been formatted and has a volume label containing “WD MY BOOK”.  It also assumes the SMTP server is called mail dot whatever the DNS domain name of the machine running the script is (i.e. my AD is jbgeek.net, so mail.jbgeek.net).  All of my client sites have a cname for their Exchange server called “mail” created in their AD DNS zone – which means I can cut and paste the same script without modification from one client site to another, which cuts down the chance of an editing error, and makes it quicker to deploy new scripts when necessary for my team.

You will need to adjust the “Get-ChildItem -Path” (lines 21 to 26) directories to fit your environment, along with the $SendEmailTo variable (line 7) and the “Start-VBRJob -Job” job name (line 41).  Line 28 defines the cutoff point for disk capacity – if the available disk space is less than 95% of the drive capacity, the backup aborts.  Line 1 is commented it out – I generally always start my scripts with # start notepad++ script.filename so that I can just cut and paste it into a command prompt to create my script file – that way as I on-board each new customer, their sites are configured the same as existing sites.  Again, it cuts down the chances of an error and makes it quicker for my guys to deploy new scripts.

**NOTE – the following deletes data from your backup cartridges. Use any tips, tricks, or scripts I post at your own risk.  I accept zero liability and responsibility if you use these scripts!!!**

Here is the command line to create the scheduled task to run as the logged in user at 10pm each weekday (it will prompt you for your password when you run it in a command prompt):

schtasks /create /tn "Weekday Veeam USB Drive Backup" /tr "\"C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe\" -ExecutionPolicy RemoteSigned -noprofile -File  C:\Windows\Run_External_Drive_Veeam_Backup.ps1" /sc daily /st 22:00:00  /rp "*" /ru "%userdomain%\%username%"

Here is the Powershell script (updated 2017.01.17 to fix an error):

# start notepad++ C:\Windows\Run_External_Drive_Veeam_Backup.ps1

add-pssnapin Veeam*

$DnsDomain = Get-WmiObject -Class Win32_NTDomain -Filter "DSDirectoryServiceFlag='True'" | Select -ExpandProperty DnsForestName
$ThisComputerName = Get-WmiObject -Class Win32_ComputerSystem | Select -ExpandProperty Name
$SupportEmailAddress = "support@$DnsDomain"

# Get-WmiObject Win32_Volume | Where-Object {$_.DriveLetter -eq $null}
$drives = Get-WmiObject -Class win32_volume | Where-Object {$_.Label -match "WD MY BOOK"}
$i = 0
Foreach($drive in $drives)
{
#Set letter
$DriveLetter = "Z:"
Set-WmiInstance -input $drive -Arguments @{DriveLetter="$DriveLetter"}
$i++
}

if (Test-Path -path Z:\){
Get-ChildItem -Path "Z:\Veeam\Backups\All VMs - Z Drive" -Include *.vbm -recurse | foreach { $_.Delete()}
Get-ChildItem -Path "Z:\Veeam\Backups\All VMs - Z Drive" -Include *.vib -recurse | foreach { $_.Delete()}
Get-ChildItem -Path "Z:\Veeam\Backups\All VMs - Z Drive" -Include *.vbk -recurse | foreach { $_.Delete()}
Get-ChildItem -Path "Z:\Veeam\Backups\All VMs" -Include *.vbm -recurse | foreach { $_.Delete()}
Get-ChildItem -Path "Z:\Veeam\Backups\All VMs" -Include *.vib -recurse | foreach { $_.Delete()}
Get-ChildItem -Path "Z:\Veeam\Backups\All VMs" -Include *.vbk -recurse | foreach { $_.Delete()}
$externaldrivelabel = Get-WmiObject -Class win32_volume | Where-Object {$_.Label -match "WD MY BOOK"} | Select -ExpandProperty label
Get-WmiObject -Class win32_volume | Where-Object {$_.Label -match "WD MY BOOK"} | % { if (($_.FreeSpace/$_.Capacity) -le '0.95' )
{Send-MailMessage -From "$($ThisComputerName.ToUpper())@$($DnsDomain.ToUpper())" -To "$SupportEmailAddress" `
-Subject "Backup Aborted due to low space on $($externaldrivelabel)." `
-Body  "Aborting nightly Veeam backup to $($externaldrivelabel) (Z:) on $($ThisComputerName.ToUpper()).$($DnsDomain.ToUpper()) due to low disk space." `
-Priority High -DNO onSuccess, onFailure -SmtpServer "mail.$DnsDomain"
break}
}
}
if (Test-Path -path Z:\){
Send-MailMessage -From "$($ThisComputerName.ToUpper())@$($DnsDomain.ToUpper())" -To "$SupportEmailAddress" `
-Subject "Now starting nightly Veeam backup to $($externaldrivelabel)." `
-Body  "Now starting nightly Veeam backup to $($externaldrivelabel) (Z:) on $($ThisComputerName.ToUpper()).$($DnsDomain.ToUpper())." `
-Priority High -DNO onSuccess, onFailure -SmtpServer "mail.$DnsDomain"
Start-VBRJob -Job "All VMs - Z Drive" -FullBackup -RunAsync
break
}else{
Send-MailMessage -From "$($ThisComputerName.ToUpper())@$($DnsDomain.ToUpper())" -To "$SupportEmailAddress" `
-Subject "Error:  No external backup drive (Z:) found on $($ThisComputerName.ToUpper()).$($DnsDomain.ToUpper())!!!" `
-Body  "Error:  Cannot find or mount the external backup drive (Z:) on $($ThisComputerName.ToUpper()).$($DnsDomain.ToUpper())!!!  Now aborting nightly backup to external hard drive!!!" `
-Priority High -DNO onSuccess, onFailure -SmtpServer "mail.$DnsDomain"
break
}

As always – Use any tips, tricks, or scripts I post at your own risk.

HOWTO: Converting from BackupExec to #Veeam when using RDX drives

Ok – I’m completely done with Backup Exec when it comes to VMware.  I’ve been selling, supporting, certified on and even using Backup Exec for our own internal backups since it was Conner Backup Exec for Windows NT 3.1, way back in 1993.  Once upon a time, it was a great product – in fact it was the only product for backups that worked worth a damn.  But it’s reliability has dropped to nothing over the past 6 or 7 years.  Technical support has been off-shored and 99.9% of the time, if I am lucky enough to finally reach someone in technical support on the phone, I can’t understand a damn word they say due to their thick accent and shitty VOIP lines crossing the Pacific Ocean.  Today was the last straw with Backup Exec, their crappy bugs, and unreliable VMware backups.  So now it’s time to fully embrace the move to Veeam, which I’ve been considering for some time (note of disclosure – I am also a certified Veeam VMCE – v7, v8, & v9)

Several of my clients have single standalone ESXi hosts, an HPE StoreOnce appliance, a physical Windows Server 2012R2 with a RDX drive or two (for offline backups), and both Backup Exec and Veeam loaded on that Windows server.  Oh – and many, many, many RDX cartridges that have months of rotated backups on them that are all three quarters full.  I can’t just erase all these cartridges in one swoop and use them for Veeam backups.  And I certainly don’t want to have to log into the clients’ servers everyday to manually delete the old Backup Exec folders off the RDX (as they come up in rotation) so that there is enough room for the nightly Veeam backup.  And finally, even though I’m dumping Backup Exec for my VMware backups, I still need to use Backup Exec to backup the 2012R2 physical instance to the same RDX cartridge that Veeam is going to use (atleast until Veeam releases their next project).  So what do I do?

A little PowerShell scripting to the rescue – that is what I am going do!

After going through a sampling of several RDX cartridges at several different client sites, I’ve determined that when Backup Exec runs with GRT enabled it dumps those backed up VMs in IMGxxxxxx folders on the root of the RDX drive (including the VMDKs).  I also discovered (or at least in the environments that I’ve setup) that GRT enabled application backups (not VMs, but rather SQL, AD, Exchange) will also be in an IMG folder with either a file called ntds.dit or edb.chk, and sometimes both!  In my case, my 2012R2 server has SQL and AD on it, so I want to be careful not to delete IMG folders that potentially contain my SQL and AD backups (which could screw Backup Exec up even more than normal when it uses that cartridge again for the 2012R2 server).

In the end, I setup the RDX drive as a new rotated drive repository in Veeam (prior to this Veeam only backed up to the HPE StoreOnce).  I then create a new Veeam job that did active fulls to the RDX drive every night (with a restore points to keep of 1).  In the job’s Advanced Settings menu, I added a pre-run script that runs C:\Windows\Remove_BackupExec_IMG_Folders.cmd.  This script in turn launches a PowerShell script that deletes all the IMGxxxxxx folders off the RDX drive except IMG folders that contain either ntds.dit or edb.chk.

**NOTE – the following deletes data from your backup cartridges. Use any tips, tricks, or scripts I post at your own risk.  I accept zero liability and responsibility if you use these scripts!!!**

Here is the contents of my batch file.

rem start notepad++ "C:\Windows\Remove_BackupExec_IMG_Folders.cmd"
PowerShell -NoProfile -ExecutionPolicy Bypass -Command "& {Start-Process PowerShell -ArgumentList '-NoProfile -ExecutionPolicy Bypass -File ""C:\Windows\Remove_BackupExec_IMG_Folders.ps1""' -Verb RunAs}"
exit/b

Here is the contents of the PowerShell script to remove the IMGxxxxxx folders (adjust the drive letter accordingly)

# start notepad++ "C:\Windows\Remove_BackupExec_IMG_Folders.ps1"
foreach ($i in Get-ChildItem R:\IMG*)
{if ((test-path "$i\ntds.dit") -eq $False -and (test-path "$i\edb.chk") -eq $False) {Remove-Item $i -force -recurse -confirm:$false}}

But wait! There is more!

Because I am still going to have to suffer with Backup Exec a while longer to backup my 2012R2 server, I need to make sure my nightly Backup Exec job doesn’t eject the RDX cartridge on me before Veeam finishes it’s RDX job.  To ensure this, I disabled the scheduled RDX jobs on my Backup Exec server.  Fortunately, Backup Exec includes a PowerShell module called BEMCLI.  So I wrote a second set of scripts as it was simply a matter of starting PowerShell from a script, importing the module, and starting the job.  So this time my scripts are a post-job script to start the Backup Exec job only after the Veeam job completes.

2017-01-11-16-33-43-snagit-0002

Here is the batch file to launch PowerShell.

rem start notepad++ "C:\Windows\Start_BE_UTIL01_RDX_JOB.cmd"
PowerShell -NoProfile -ExecutionPolicy Bypass -Command "& {Start-Process PowerShell -ArgumentList '-NoProfile -ExecutionPolicy Bypass -File ""C:\Windows\Start_BE_UTIL01_RDX_JOB.ps1""' -Verb RunAs}"
exit/b

And here is the PowerShell script to start the Backup Exec job called “23:10 UTIL01 RDX-Full”.

# start notepad++ "C:\Windows\Start_BE_UTIL01_RDX_JOB.ps1"
Import-Module BEMCLI
Get-BEJob -Name "23:10 UTIL01 RDX-Full" | Start-BEJob -confirm:$false

Now when my Veeam backup job to RDX starts, it deletes all the IMGxxxxxx folders off the RDX drive (unless those folders contain either ntds.dit or edb.chk), and when it completes, it starts the remaining Backup Exec job, which ultimately ejects the RDX cartridge when it completes.

As always – Use any tips, tricks, or scripts I post at your own risk.

HOWTO: Install the Veeam One v9.5 Monitor Client on a Domain Controller

Most of my client sites are smaller companies, with one, two or three hypervisor hosts and a single Windows 2012R2 physical installation to manage the hypervisor cluster, shared storage, and backups.  Typically we configure the 2012R2 physical installation as a domain controller, in addition to a pair of virtualized domain controllers on the hypervisor cluster.  We generally utilize Veeam Availability Suite for backup and cluster monitoring, with Veeam Backup & Replication being installed the 2012R2 physical instance and Veeam One being installed in a virtual machine.  When we need to manage something in the customer’s environment, we typically RDP the 2012R2 physical instance and do whatever we need from there.  This means we install all our management tools on there whenever we can.

This approach has always worked well for us except for when vendors decide to prevent installation of their software on domain controllers.  Now I won’t dispute that some software packages definitely deserved to be blocked from installing on a domain controller, but in this particular case, Veeam has decided that the Veeam One Monitor client is just too dangerous for a domain controller – huh?  Why?  It is just a client (no different than the vSphere vCenter client, or the Vertias Backup Exec console utility) that connects to another server where the software resides.  So what do you do when you still want to install the Veeam One v9.5 Monitor Client on a domain controller?  You edit the installer.  🙂

In Veeam One version 8 and 9, you needed to edit the veeam_monitor_cln_x64.msi installer and change the VmACheckIsDC entries in both the InstallExecuteSequence and InstallUISequence tables to True, then save the installer.  Then you could run the installer on your domain controller.

In Veeam One v9.5 however, Veeam has changed their checking logic, so the above steps no longer work.  The good news is that with v9.5, you only need to modify one table entry now instead of two.  So here are the necessary steps.

  1. Extract the VeeamONE.9.5.0.3201 iso image to a folder, then navigate to the extracted Monitor folder.
  2. Copy veeam_monitor_cln_x64.msi to veeam_monitor_cln_x64_dc_installer.msi
  3. Open veeam_monitor_cln_x64_dc_installer.msi with an msi editor such as Panataray’s SuperOrca
  4. Scroll down to the VMLaunchConditions table
  5. There you should find three entries, and one of them will have the following data: “MsiNTProductType <> 22016-11-21-10-39-00-snagit-0048
  6. Edit the entry and change “<>” to “=” so you end up with “MsiNTProductType = 22016-11-21-10-38-46-snagit-0047
  7. Save the msi and close your msi editor
  8. On your domain controller, launch the veeam_monitor_cln_x64_dc_installer.msi to install the Veeam One v9.5 Monitor client

As always – Use any tips, tricks, or scripts I post at your own risk.