As of late I have been running into a few issues with VMware vSphere and vVOLs. I could go on for hours about my hatred of SSL certificates and how much of a pain that they have been for me in the past. But that is not what you are here for (at least I hope not).
This post is going to be a little bit different from my normal “How To” posts that I normally do. In this article I will focus more on the things that I have learned, and steps that need to be taken when working with VMware’s vSphere and vVOLs.
Here is a little back story on what a vVOL is and how it integrates with vSphere. To be completely honest with you, I am not an expert on the subject by any means. I would say that everything that I learned from this technology is either self taught or from VMware Support. Speaking of VMware Support, on numerous occasions I have been told by their Level 2 and 3 storage engineers that they know very little about how it actually works. This actually makes sense and let me explain.
A vVOL or Virtual Volume is a block of storage that gets presented from the storage provider (SAN usually) and vCenter just connects to it as a subscriber. From there vCenter replicates the connections to the ESXi hosts which these connections are independent of the vCenter connection. All of the actual provisioning and managent is actually done on the storage array. So unlike a VMDK where you have to mount the LUN and then format it. A vVOL gets connected to vCenter by via the VASA Provider screen and create a vVOL in vCenter which then gets “trickled” down to the ESXi hosts that you want to be able to see it.
Now here is where the tricky parts come in. At least in the case of Pure Storage only 1 vCenter may connect to a particular Storage controller at one time unless you are using a CA issued certificate. Which lets face it, half the time IT teams barely use DNS and you expect them to use SSL certs if it is not client facing. The good news is, Pure thought of this and that is why when you connect to the storage controller from vCenter the storage controller will issue a self signed certificate which will be used for the connection. The bad news is, the SSL cert is only good for a year and vCenter starts to alert that the certificate is expiring after 100 days or something like that.
As long as everything is good connectivity wise, renewing the certificate is as easy as logging into vCenter >clicking on the vCenter object > going to configure and clicking on storage providers > finding your Certificate and clicking refresh. From there vCenter sends a request to the storage controller and the storage controller will generate a new certificate and send it to vCenter and that is it.
Well sort of, see if it was that easy, I would not be writing this post. A few months back I had gone through the above process and it worked just fine. All was well for about 2 weeks, and then I started getting alerts (little yellow triangles in the vCenter console) saying that the storage paths to the vVOLs lost their redundancy. I opened a case with VMware support and they looked at it and said it was not a them issue it was an issue with the storage. Now to cut a long support story short, my case ended up getting routed to the VMware certificates team after my hosts started disconnecting from storage and I had to escallate the case to a P1.
The Certificates engineer saw something in the logs saying there was a certificates mismatch and wondered if that could be part of the issue. By that point things had gone too far and the fix was to delete the storage provider and add it back in manually and that brought it back up. However a few months after that on a completly different vCenter I started to see the same thing. I again opened a case and got the same engineer as before. They showed me a little trick which is to right click on the hosts that are connected to the vVOLs, go to Certificates, and click renew certificates and refresh CA certificates and that fixed the issue!!
So here is my understanding of what happened here. Even though I refreshed the vVOL certificate on the vCenter, something happened which did not allow the certifcate to propagate down to all the vCenters which required manual intervention before the ESXi hosts terminated their connections with the vVOLs.
So now going forward my process is to renew the Certificate from the VASA Providers > Refresh the host Certificates > and Renew the CA certificates on the ESXi hosts.
I hope you found this post helpful and I apologize for it being so wordy. Hopefully this will help someone in the future which is the whole purpose of this blog.
