Categories
Virtualization Vmware

vCenter – Migrate is greyed out

Intro

Recently I was working on an issue where the option to migrate was greyed out inside of VMware vCenter. At the time we were not sure how this issue started, but we eventually figured it out. We were using a snapshot based backup solution which had an issue.

Allow me to explan the process. We learned that when a VM gets snapshotted, the VM gets locked by vCenter. This is to prevent the VM from moving while the snapshot is being created. I was never sure how this “lock” actually worked, but I learned that the VM id gets added to the VPX_Disabled_Methods table. Once the Snapshot get created, vCenter is supposed to remove the id from the table which will then allow you to perform migrations.

In our case, what happened was the backup created the snapshot, but never told vCenter to remove the lock. So after a bunch of calls with VMware we learnd about the VPX_Disabled_Methods table.

Warning

NOTE: Do not run this command unless you know what you are doing!!! However, running a select command should not cause any issues….

How to fix the issue

First you will need to log into the vCenter appliance console using SSH as root.

You will then need to run the following command to access the VCDB (vCenter Database)

sudo /opt/vmware/vpostgres/1.0/bin/psql -d VCDB

To see if you have any VMs that are impacted you can run the following command.

select * from VPX_DISABLED_METHODS;

If any VMs are impacted then you should see something in the list. If you do, you can take that vm-id and look them against vCenter which will probably be having the migration issue.

In order to resolve the issue you will need to delete the VM ID from the table. To do this you should run a command like this

select * from VPX_DISABLED_METHODS WHERE ENTITY_MO_ID_VAL = 'vm-xxx;

Or if you are like me, and don’t want any of your VMs to be locked, you can be like me and do the sledge hammer approach

delete * from VPX_DISABLED_METHODS;

This will clear all of the entries from the table.

Closing

I hope you found this post helpful, and if so please share with your friends.

Categories
Uncategorized Virtualization Vmware

VMware vCenter vVOLs Host Certificate issues

As of late I have been running into a few issues with VMware vSphere and vVOLs. I could go on for hours about my hatred of SSL certificates and how much of a pain that they have been for me in the past. But that is not what you are here for (at least I hope not).

This post is going to be a little bit different from my normal “How To” posts that I normally do. In this article I will focus more on the things that I have learned, and steps that need to be taken when working with VMware’s vSphere and vVOLs.

Here is a little back story on what a vVOL is and how it integrates with vSphere. To be completely honest with you, I am not an expert on the subject by any means. I would say that everything that I learned from this technology is either self taught or from VMware Support. Speaking of VMware Support, on numerous occasions I have been told by their Level 2 and 3 storage engineers that they know very little about how it actually works. This actually makes sense and let me explain.

A vVOL or Virtual Volume is a block of storage that gets presented from the storage provider (SAN usually) and vCenter just connects to it as a subscriber. From there vCenter replicates the connections to the ESXi hosts which these connections are independent of the vCenter connection. All of the actual provisioning and managent is actually done on the storage array. So unlike a VMDK where you have to mount the LUN and then format it. A vVOL gets connected to vCenter by via the VASA Provider screen and create a vVOL in vCenter which then gets “trickled” down to the ESXi hosts that you want to be able to see it.

Now here is where the tricky parts come in. At least in the case of Pure Storage only 1 vCenter may connect to a particular Storage controller at one time unless you are using a CA issued certificate. Which lets face it, half the time IT teams barely use DNS and you expect them to use SSL certs if it is not client facing. The good news is, Pure thought of this and that is why when you connect to the storage controller from vCenter the storage controller will issue a self signed certificate which will be used for the connection. The bad news is, the SSL cert is only good for a year and vCenter starts to alert that the certificate is expiring after 100 days or something like that.

As long as everything is good connectivity wise, renewing the certificate is as easy as logging into vCenter >clicking on the vCenter object > going to configure and clicking on storage providers > finding your Certificate and clicking refresh. From there vCenter sends a request to the storage controller and the storage controller will generate a new certificate and send it to vCenter and that is it.

Well sort of, see if it was that easy, I would not be writing this post. A few months back I had gone through the above process and it worked just fine. All was well for about 2 weeks, and then I started getting alerts (little yellow triangles in the vCenter console) saying that the storage paths to the vVOLs lost their redundancy. I opened a case with VMware support and they looked at it and said it was not a them issue it was an issue with the storage. Now to cut a long support story short, my case ended up getting routed to the VMware certificates team after my hosts started disconnecting from storage and I had to escallate the case to a P1.

The Certificates engineer saw something in the logs saying there was a certificates mismatch and wondered if that could be part of the issue. By that point things had gone too far and the fix was to delete the storage provider and add it back in manually and that brought it back up. However a few months after that on a completly different vCenter I started to see the same thing. I again opened a case and got the same engineer as before. They showed me a little trick which is to right click on the hosts that are connected to the vVOLs, go to Certificates, and click renew certificates and refresh CA certificates and that fixed the issue!!

So here is my understanding of what happened here. Even though I refreshed the vVOL certificate on the vCenter, something happened which did not allow the certifcate to propagate down to all the vCenters which required manual intervention before the ESXi hosts terminated their connections with the vVOLs.

So now going forward my process is to renew the Certificate from the VASA Providers > Refresh the host Certificates > and Renew the CA certificates on the ESXi hosts.

I hope you found this post helpful and I apologize for it being so wordy. Hopefully this will help someone in the future which is the whole purpose of this blog.

Categories
Uncategorized Virtualization Vmware

vCenter – Unable to login with WinSCP

Hey all it has certainly been a while since I have posted anything, but I am back now and I have quite a few things in the pipeline including a walk through of my new home lab!! For now lets get down to business with being unable to login with WinSCP.

Today I wanted to cover an issue that has been plaguing me quite a bit the last few months. As I am sure you are all aware, VMware vSphere and ESX 6.7 has been out of support since October 2022. Back in August of 2022, my team members at the company I work for worked dilligently to upgrade all of the vCenters and ESXi hosts in out environment.

If you are not aware, upgrading a VCSA appliance from 6.7 to 7.0 involves deploying a new 7.0 appliance and migrating your settings from one to another. Our Friends at VMware have done an amazing job with building a wizard that does most of the work for you. Perhaps as part of my new lab I will try do a guide on the upgrade process as I am sure there are many companies out there who have yet to do the upgrade. In fact I might even go as far as upgrading to 8.0!! Tune in for more.

Back to the matter at hand, When you deploy a new vCenter one of the issues you may run into is not being able to log in with WinSCP. For those of you that do not know, WinSCP is a free application (donations are accepted and appreacated) that allows you to transfer files using the SCP protocol (among others) on a Windows PC and it gives you a nice graphical user interface. You can download a copy of WinSCP here. You can also download it using Chocolatey if that is your thing.

When you try to connect to vCenter using WinSCP you may get this message:

Host is not communicating for more than 15 seconds. If the problem repeats, try turning off ‘Optimize connection buffer size’.

This can be particularly annoying when you need to add or remove a file from vCenter and because like me you probably don’t add or remove files from vCenter all that often so most times you forget that this could be an issue.

To solve the issue you will need to log into vCenter using SSH as root and run this command:

chsh -s /bin/bash root

and press enter.

Once you have done this you can then retry connecting via WinSCP and the issue should be resolved. If you don’t have the root credentals, please check out this article to find out how you can get around that.

Categories
Uncategorized Virtualization Vmware

How to enable BASH shell on vCenter 6.0

According to VMware’s KB articles they claim that this settings change is no longer needed when logging into VCSA Appliances in 6.5 – 7.0. However I am not 100% certain that that is true as, to be honest I always log in as root and have never had to do it. Do have some collegues who claim that they need to do this when they are using their AD Authenticated credentials.

While logged into vCenter as you not root account you will need to type (or copy and paste) this into your SSH session:

shell.set –enable True

In reality, you could also just log into the VAMI and start the BASH service there which I think is what I normally do anyway and thus why I have never had to run this command.

Categories
Virtualization Vmware

vCenter 6.5 – Unable to Authenticate

Happy Monday everyone! I hope you all had a great weekend and Holiday. Guess what happened to me this morning? I came into the office (virtually of course) and discovered that our VDI vCenter environment was inaccessible. Now this is an older environment on its way out the door so I am not too concerned with it. However it would make our lives easier to have vCenter working for the decommissioning process.

When I tried to log into this VCSA Server I would receive the following message in a red banner across the top of the screen:

[400] An error occurred while processing the authentication response from the vCenter Single Sign-On server. Details: ‘Trusted root certificates’ value should not be empty.

When I said this environment is old, it was upgraded from 5.5 and is now comprised of a VCSA appliance and an external PSC. Although the database has been migrated to a vPostgress DB instead of the SQL database that is started on, we have still noticed issues as time has gone one.

We tried to log in both with AD credentials and the local administrator user and both failed. Which was very concerning as we did have an issue with a DNS a few months back which knocked out AD authentication and all of our hosts in a different environment. Resolving that issue required a ticket to be opened with VMware.

I logged into the VAMI on both the PSC and the vCenter server (AD Authentication on both) and I did not see any issues there. Everything looked health and good. Since this environment is going away we decided to see if a good old reboot would fix it since we could not remember when the last reboot on this system occurred.

After rebooting the VCSA appliance everything seemed okay. In fact the vCenter console logged me in automatically. We plan to look into this more, however with this environment being decommissioned in the next few days I don’t think we will find anything of value.

If I do find a root cause for this one I will make sure to post a update on this.

Categories
Home Lab PowerCLI Starwind Virtual SAN Virtualization Vmware

Home Lab 2.0 – Starwind VSAN : The Greatest addition to my Home Lab

At the end of my previous post about The Beast I mentioned about lacking shared storage for my Virtual Hosts. My first thought was to go out and purchase a NAS like a Synology or a QNAP NAS. However I am trying to do this on the cheap and an extra $1200 when figuring in the cost of drives is just not in my budget. Now I am not sure about Synology, but I believe that they are comparable to QNAP in the sense that you can present disks as iSCSI LUN. Which I have done several times in the past and works brilliantly.

So here I am sitting in my office looking at the Beast and I was racking my brain trying to figure out what I can do with all this storage in this one box and how I can share it among my Virtual Hosts. I started looking at FreeNAS and running it as a VM in VMware Workstation and that worked! Until I had a power failure and the virtual disks became corrupted and I could not get them to come back online. Very Frustrating, my search continued.

Anyway, after doing some research I discovered Starwind VSAN. It allows you to take storage on a Windows machine and present it as an iSCSI LUN. I thought this is great and I signed up for a trial which gives you the ability to install it on one host but no graphical interface (it uses PowerShell) but unlimited LUNs. After some quick googling about the syntax I was able t piece together the commands with relative ease. Within a few minutes I was up an running with three new iSCSI LUNs and they were mounted in vCenter.

I mentioned earlier about having issues with FreeNAS and power failures, well ironically we had a power failure but Starwind VSAN came back online without a hitch. I was very impressed with how resilient it was. I have not had the chance to really put it through it’s paces, but so far I think I found a winner.

I plan to s follow up post once I really get off and running to the races so stay tuned for updates.

Categories
Home Lab Virtualization Vmware

Home Lab 2.0 – The Beast

Continuing on our series on my Home lab we now move into present day. Well…. a few months ago at least. As I mentioned in my previous post about my Razor Blade 15, I used that as my home lab until I was able to procure parts for my current hyper converged VMware Home Lab.

Below is my part list for The Beast:

Once all the parts came in I managed to get it all hooked up relatively quickly and without issue. I must admit that this is the first time I have ever used a Noctua Cooler, but it is whisper quiet, and nowhere near as expensive as I thought it would be.

So moving away from the hardware I then installed Window 10 and after 4 hours or so of Windows updates I was finally able to install VMware Workstation Pro 15. Well, almost…. it turns out that Virtualization Support was not enabled by default. But I can get into that at a later time. I then built out a Domain Controller (another 4 hours of Windows Updates) and 3 Virtual hosts (no Windows Updates). Once the hosts were built I then deployed a VCSA appliance.

Now I think I have everything I will need to play with VMware Products. That is until I remembered I don’t have Shared Storage for my Virtual hosts. What ever will I do?!

Stay tuned for my next post in this series, especially if you are looking to build a home lab on the cheap.

I hope you found this helpful and if so please share it with your friends, we could really use your support.

Categories
Virtualization Vmware

Whats new in Pricing, Metering, and Reporting:…

Whats new in Pricing, Metering, and Reporting: June 2020 Update

Whats new in Pricing, Metering, and Reporting:…

We just started the first webinar in the “Whats new in Cloud Provider Pricing, Metering, and Reporting” series. In this blog we’ll visit what was discussed and as well as provide the action items that partners can perform today to plan ahead or make monthly reporting easier. We’ll cover what’s […]


VMware Social Media Advocacy

Categories
Virtualization Vmware

DevOps: Culture – Collaboration, Empowerment,…

DevOps: Culture – Collaboration, Empowerment, Autonomy

DevOps: Culture – Collaboration, Empowerment,…

Organizations that have successfully transformed their technology delivery through DevOps consistently recognize that it requires much more than tool and process change. We repeatedly find that leading indicators for DevOps success correlate directly with cultural changes required to support […]


VMware Social Media Advocacy

Categories
Virtualization Vmware

Kubernetes as a Service

Kubernetes as a Service

Kubernetes as a Service. How you can request, deploy and manage Kubernetes Clusters and Namespaces with vRealize Automation using the self-service catalog.


VMware Social Media Advocacy