Manually removing ghost vVols from IBM SVC-based storage

As part of my evaluation of presenting vVols to vCenter from an IBM FlashSystem V9000, I decided to start from scratch after learning a bit about the benefits and limitations of the system. That is: I like vVols a lot, but I learned some things in my tests that I wanted to do differently in actual production.

Unfortunately, once I had migrated my VMs off the vVol datastores, I still couldn’t detach the relevant storage resources from the storage service in Spectrum Control Base. The error message was clear enough: I’m not allowed to remove a storage resource that still has vVols on it. My frustration was based in the fact that vCenter showed no VMs nor files on any of the vVol datastores, but I could clearly see them (labeled as “volume copies”) in the “Volumes by Pool” section in the SVC webUI on the V9000.

At least as of version 7.6.x of the SVC firmware, there is no way of manually removing vVols from the GUI, and as usual in such cases, we turn to the CLI:
I connected to the V9000 using ssh, taking care to log on as my VASA user. All virtual disks on the V9000 can be listed using the lsvdisk command. The first two columns are their ID and name, and any of these parameters can be fed to the rmvdisk command to manually remove a volume.

Just to be clear: The rmvdisk command DELETES stuff. Do not use it unless you really mean it! With that warning out of the way; once I had removed the volumes and waited a couple of minutes for the change to propagate to Spectrum Control Base, detaching storage resources from storage services was a breeze.

VMware Storage Providers and Certificate issues

While trying to test out vVols in our vSphere 6.5 environment, presented via IBM Spectrum Control Base 3.2 from a StoreWize V9000 SAN, I ran into a small issue that took me a while to figure out:

I installed Spectrum Control Base 3.2 and presented its web services via a FQDN.
To avoid the nagging of modern browsers, I used a regular wildcard certificate valid for the domain I chose to use.
After the initial setup, when I tried to add SCB as a storage provider in VMware, I got the following error message: “A problem was encountered while provisioning a VMware Certificate Authority (VMCA) signed certificate for the provider.
A web search showed me that this was a pretty common problem with several VASA providers, but none of the suggested solutions applied to our environment. After half an hour of skimming forums and documentation I found the following quote in an ancient support document from VMware:
Note: VMware does not support the use of wildcard certificates.

So: I generated a self-signed certificate in the Spectrum Control Base server webUI, and the problem disappeared.

Lesson of today: We don’t use wildcard certificates in a VMware service context.