Categories
Virtualization Vmware

vCenter – Migrate is greyed out

Intro

Recently I was working on an issue where the option to migrate was greyed out inside of VMware vCenter. At the time we were not sure how this issue started, but we eventually figured it out. We were using a snapshot based backup solution which had an issue.

Allow me to explan the process. We learned that when a VM gets snapshotted, the VM gets locked by vCenter. This is to prevent the VM from moving while the snapshot is being created. I was never sure how this “lock” actually worked, but I learned that the VM id gets added to the VPX_Disabled_Methods table. Once the Snapshot get created, vCenter is supposed to remove the id from the table which will then allow you to perform migrations.

In our case, what happened was the backup created the snapshot, but never told vCenter to remove the lock. So after a bunch of calls with VMware we learnd about the VPX_Disabled_Methods table.

Warning

NOTE: Do not run this command unless you know what you are doing!!! However, running a select command should not cause any issues….

How to fix the issue

First you will need to log into the vCenter appliance console using SSH as root.

You will then need to run the following command to access the VCDB (vCenter Database)

sudo /opt/vmware/vpostgres/1.0/bin/psql -d VCDB

To see if you have any VMs that are impacted you can run the following command.

select * from VPX_DISABLED_METHODS;

If any VMs are impacted then you should see something in the list. If you do, you can take that vm-id and look them against vCenter which will probably be having the migration issue.

In order to resolve the issue you will need to delete the VM ID from the table. To do this you should run a command like this

select * from VPX_DISABLED_METHODS WHERE ENTITY_MO_ID_VAL = 'vm-xxx;

Or if you are like me, and don’t want any of your VMs to be locked, you can be like me and do the sledge hammer approach

delete * from VPX_DISABLED_METHODS;

This will clear all of the entries from the table.

Closing

I hope you found this post helpful, and if so please share with your friends.

Categories
Uncategorized Virtualization Vmware

vCenter – Unable to login with WinSCP

Hey all it has certainly been a while since I have posted anything, but I am back now and I have quite a few things in the pipeline including a walk through of my new home lab!! For now lets get down to business with being unable to login with WinSCP.

Today I wanted to cover an issue that has been plaguing me quite a bit the last few months. As I am sure you are all aware, VMware vSphere and ESX 6.7 has been out of support since October 2022. Back in August of 2022, my team members at the company I work for worked dilligently to upgrade all of the vCenters and ESXi hosts in out environment.

If you are not aware, upgrading a VCSA appliance from 6.7 to 7.0 involves deploying a new 7.0 appliance and migrating your settings from one to another. Our Friends at VMware have done an amazing job with building a wizard that does most of the work for you. Perhaps as part of my new lab I will try do a guide on the upgrade process as I am sure there are many companies out there who have yet to do the upgrade. In fact I might even go as far as upgrading to 8.0!! Tune in for more.

Back to the matter at hand, When you deploy a new vCenter one of the issues you may run into is not being able to log in with WinSCP. For those of you that do not know, WinSCP is a free application (donations are accepted and appreacated) that allows you to transfer files using the SCP protocol (among others) on a Windows PC and it gives you a nice graphical user interface. You can download a copy of WinSCP here. You can also download it using Chocolatey if that is your thing.

When you try to connect to vCenter using WinSCP you may get this message:

Host is not communicating for more than 15 seconds. If the problem repeats, try turning off ‘Optimize connection buffer size’.

This can be particularly annoying when you need to add or remove a file from vCenter and because like me you probably don’t add or remove files from vCenter all that often so most times you forget that this could be an issue.

To solve the issue you will need to log into vCenter using SSH as root and run this command:

chsh -s /bin/bash root

and press enter.

Once you have done this you can then retry connecting via WinSCP and the issue should be resolved. If you don’t have the root credentals, please check out this article to find out how you can get around that.

Categories
Uncategorized Virtualization Vmware

How to enable BASH shell on vCenter 6.0

According to VMware’s KB articles they claim that this settings change is no longer needed when logging into VCSA Appliances in 6.5 – 7.0. However I am not 100% certain that that is true as, to be honest I always log in as root and have never had to do it. Do have some collegues who claim that they need to do this when they are using their AD Authenticated credentials.

While logged into vCenter as you not root account you will need to type (or copy and paste) this into your SSH session:

shell.set –enable True

In reality, you could also just log into the VAMI and start the BASH service there which I think is what I normally do anyway and thus why I have never had to run this command.

Categories
PowerCLI Virtualization Vmware

PowerCLI – Datastore Report

I recently was working in an environment where they are still running vSphere 6.0. While doing some storage expansions I discovered that the Flash Client and the C# Client were not reporting the same size on the Datastore. This can be very frustrating as you are not sure which one you can trust. So as a tie breaker I decided to turn to my good friend PowerCLI.

It turns out that the PowerCLI and the C# client were reporting the same thing. So I decided to create a PowerCLI Script to assist me in my work. You can find the script there on my Github:

https://github.com/kenbshinn/PowerCLI-Scripts/blob/master/VM_Datastore_Report.ps1

I hope you found this post helpful and will share it with your friends.

Categories
Virtualization Vmware

How to unlock and reset SSO password in vSphere 6.x (2146224)

So I have a bit of a embarrassing confession to make. I forgot to record the Administrator password for my VCSA Appliance. Total disclosure, I was freaking out and I really thought I was going to have to start from scratch. I did some research I was surprised to find out that you can actually reset the Administrator account on a VCSA appliance as long as you have the root password for the appliance and you have access to the VCSA Console. Below are a list of the links to the KB Articles from VMware.

Resetting SSO Administrator Password
https://kb.vmware.com/s/article/2034608
Resetting SSO Administrator – VCSA 6.x

Below is the PUTTY session as an example.

[email protected]:~$ ssh [email protected]
ssh: Could not resolve hostname devvcsa01.xxx.xxxxx: Name or service not known
[email protected]:~$ ssh [email protected]
The authenticity of host ‘172.26.44.18 (172.26.44.18)’ can’t be established.
ECDSA key fingerprint is SHA256:7E4K1HVpg2ExWz+vEkkRdJ0M5jUYftb3HZw6OSDKFEICSOEPWWKYERe4.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added ‘172.26.44.18’ (ECDSA) to the list of known hosts.

VMware vCenter Server Appliance 6.5.0.21000

Type: vCenter Server with an embedded Platform Services Controller

Password:
Connected to service

    * List APIs: “help api list”
    * List Plugins: “help pi list”
    * Launch BASH: “shell”

Command> shell.set –enabled true
Command> shell
Shell access is granted to root
[email protected] [ ~ ]# /usr/lib/vmware-vmdir/bin/vdcadmintool

==================
Please select:
0. exit
1. Test LDAP connectivity
2. Force start replication cycle
3. Reset account password
4. Set log level and mask
5. Set vmdir state
6. Get vmdir state
7. Get vmdir log level and mask
==================

3
  Please enter account UPN : [email protected]
New password is –
/a+p|8M?vRl`%”p4*+oZ

==================
Please select:
0. exit
1. Test LDAP connectivity
2. Force start replication cycle
3. Reset account password
4. Set log level and mask
5. Set vmdir state
6. Get vmdir state
7. Get vmdir log level and mask
==================

Once you go through all these steps you are now able to log into VCSA with that temporary password that you are given and you are also able to reset it as well.

I hope you find this post helpful, and if you do please share it out to your friends.

Categories
PowerCLI Virtualization Vmware

Power off entire Virtual Environment using a PowerShell Script and PowerCLI

So I have a lab that I manage where we have had several scheduled power outages in the last few months. So it is my job to make sure that we power off the lab so none of our equipment have issues when we bring it back up.

I would estimate that our lab is comprised of about 90% VMware ESXi Hosts, and after going through the exercise of powering off the whole lab 1 or 2 times, it became it bit of a pain.

So I created a PowerShell Script using PowerCLI to not only power off all the Virtual Machine, but also the Virtual Hosts, and the VCSA appliance itself.

I have posted the script to my Github which you an check out here:

https://github.com/kenbshinn/Poweroff_VirtualEnvironment

Feel free to check it out and let me know what you think.

I hope you found this post helpful, and if so, please share it with your friends.

Categories
Virtualization Vmware

All vCenters not showing up after adding a new one to an SSO Domain

As I have mentioned before, the company I work for had a disaster event that took place almost a year ago and because of that we have had  some of our infrastructure duct taped together.

Today I am proud to say that I had the opportunity to rip off another piece of that duct tape and actually move our Virtual Infrastructure forward.

You see, Pre-Disaster we had a single vCenter appliance which managed 3 sites. (Yes I know… yuck)

But, because of the disaster we had to move all of our services from the 1 site to the other 2. In the middle of the DR event I had to create 2 VCSA appliances to be able to manage the 2 sites, and due to a lack of sufficient network connectivity at the time, they were just islands. I even set them up a separate SSO Domains.

Fast forward to today, and I have now consolidated these 2 SSO Domains down to 1 and I must say it is pretty slick.

I did however run into a bit of an anomaly, which is the purpose of my post today. You see on the VCSA appliance that was added to the existing SSO domain, I discovered that I could see the first VCSA Appliance in the vSphere Web Client as well as it’s inventory which was awesome!

However on the vSphere Web Client of the Original VCSA Appliance, I can only see the 1 VCSA Appliance.

I consulted Dr. Google but found nothing at first, until I came across this post on the IBM Cloud for VMware Solutions site.

It turns out that you need to restart the vSphere Web Client in order for the new vCenter server to appear.

Just encase the link dies I will post the resolution here:

This is a known VMware 6.5 issue.

To resolve the problem, you must restart the vSphere Web Client:

Using the root account, connect over ssh to the vCenter VM (virtual machine) of the previously ordered instance.
Type shell to enter the bash shell.
Enter service-control –stop vsphere-client to stop the client.
Enter service-control –start vsphere-client to restart the client.
After the vSphere Web Client of the previously ordered instance is restarted, confirm that the vCenter Server system for the newly added secondary instance is visible in the vSphere Web Client.

NOTE: Rebooting the VCSA Appliance will also resolve your issue.

I hope you found this helpful, and if so please let me know and share with your friends.

Categories
Virtualization Vmware

Unable to log into a new VCSA Appliance added to an SSO Domain

So today I ran into an interesting issue. I was adding a new VCSA appliance to an preexisting SSO domain for a post that will come out in the coming weeks and I discovered a few things.

First of all, I learned that there are not many blogs covering this so I figured I would make a post just encase I get hit with a brick and forget.

So I ran through the install of the new VCSA appliance and selected all the settings as you would normally do. However when the installation completed it said everything was fine and I should be able to log in at the new URL.

I opened the URL and tried to log in…. and It failed. This being the first time I have tried this I was not sure what to expect, and here is what I mean:

The Platform Services Controller (or PSC) is responsible for authentication to vSphere. That being said I figured I might be able to log in with my Domain credential…. I was wrong.

So I then tried logging in with the Administrator Credentials for the SSO Domain…. that too did not work.

I then logged into the vSphere Web Client for the First VCSA to see if I could see what is going on over there. It turns out, from an SSO Perspective, everything was running great. I went to see if I could reset the Administrator password and discovered that I must not have had enough coffee this morning. You see, because vSphere is based on some form of Linux Kernel, I forgot to take into account that THE USER NAME IS CASE SENSITIVE!!!!!

So I went back to the login screen and logged in with the correct credentials and IT WORKED!!

Now to figure out why AD Authentication was not working…..

Well it turns out that that lack of coffee I mentioned earlier came back to bite me yet again <face palm>

Although the SSO component was working and showed my AD domain as a valid and default authentication source, it doesn’t mean squat if the server is not a member of the Active Directory Domain. A quick add to AD and a reboot later and we were in business.

Lesson Learned:

1. Don’t skip out on Coffee
2. Linux based credentials are Case Sensitive
3. Drink more Coffee
4. Make sure that host is added to AD before you try to authenticate.

I hope you found this helpful and if I ever get around to it I will add screen shots for a more TL:DR experience.

Categories
Tools Virtualization Vmware

Tools Discussion – VMWare Converter Standalone

Today I wanted to talk about VMware Converter Standalone which has been a constant in my bag of tricks for a long time. VMware Converter is a tool that allows you to convert a Physical Machine to a Virtual Machine which can run on a VMware ESXi host or VMware Workstation / Player. You can also use VMware Converter to resize Virtual machines or even convert them from Thick Provisioned virtual disks to thin.

Before VMware Converter was a free Standalone product, it was an add-on to vCenter (version 3-4.1 if my memory was correct) that was licensed by VMware. It enabled you (much like it does now) to convert a physical machine and turn it into a Virtual Machine. Pretty Sweet huh?!

Somewhere between vSphere 4 and 5 they decided to remove it from vCenter as an add on and make it a a separate product and best of all made it free!

During the installation process you are now asked to decide if you are doing a standalone installation or a Client Server install. I will probably do an install guide for this later so I can discuss further as both have their merits.

If you are interested in getting a copy of VMware Converter Standalone you can follow this link. It will also be available on the Tools page once this post goes live.

I hope you found this post helpful and please check back later for more articles.

Categories
Virtualization Vmware

How to convert VMDK from Thick to Thin Provision

As I have stated in a previous post, I have been trying to help out on the VMware Community Forum. One of the threads that I was able to help out on was how to convert a VMDK from thick to thin.

I have had the opportunity to do this several times in the past, but I have never needed to document. I was able to do a step by step procedure from memory for the thread, but I figured doing an actual guide might be helpful to someone else. 
The first thing that you need to realize is that the VM you want to change from Thick Provision to Thin MUST be powered off before you can convert it. 
Launch the VMware vCenter Converter Standalone Client software.
Click on the Convert Machine button.

That will launch the Converter wizard

By Default  the Source machine will be set to Powered on VM. You will need to select the Powered off radio button which will give you to option to select VMware Virtual Machine. 
Enter the Server Name or IP address, Username, and Password for the Source system and click Next

This will cause the Wizard to do a check against the Source machine to make sure that the credentials are able to access the vCenter server. 

Once you are logged in, will see the your vCenter server and ESXi hosts. Click on the vCenter server, or ESXi Host to see all the VMs that are available. 

Select the Powered off VM you wish to convert to thin provisioned and click Next. 

Next you will need to enter the Server Name, User name and Password for the destination host. Before you ask, yes the source and destination host can be the same server, however the Virtual Machine name will need to be different. 
Click Next to Continue

As I mentioned above, you will need to give the Virtual Machine a unique name, otherwise vCenter gets pretty mad. 
Select the Data Center you would like to put the VM on and click Next. 

Select the Host, Datastore, Virtual Hardware Version, and click Next.
NOTE: You should probably pick a different datastore that the one you used before, just to make sure that you don’t run out of space. 

This will take you to the Options screen which is where you are allowed to change the Virtual Disk from Thick to Thin. 

Click Edit as seen in the screen below.

This will bring up the disk configuration for this Virtual Machine.

Click on the drop down under Type and change it from Thick to Thin and then click Next.

You will then be presented with the Summary screen which you can review, and then click Finish to start the conversion. 

 As you can see the Job gets submitted and will run until it completes successfully

Once it completes you will see in vCenter that there is a new Virtual Machine and if you check the settings of that Virtual Machine the VMDK will be Thin Provisioned. 
Thank you for stopping by, and I hope you found this post helpful and please check back later for more updates.