To start, let me apologize for not taking screen shots to share this issue. If this issue occurs in the future I will be sure to take them then and fill in the blanks.
Today after coming into the office from a long weekend because of Memorial Day we discovered that our Lab had an issue on Friday at 7 PM. While I was working on getting everything backup and online I found that my precision Exchange 2013 Server that I built in a previous post was showing up as being grayed out.
Play ominous music
My knee jerk reaction here was to remove the VM from inventory and try to re-add the VM back into inventory by right clicking on the .vmx file and click add to inventory. Something I have done hundreds of times before without issue. However this time was different, the .VMX file would not allow me to add the VM to Inventory. I also noticed there were more than 1. VMX file.
Fortunetly I was already logged into the host as root and saw the the VM was still in inventory. Then I looked at the other host while logged in as root and saw the same thing. I then processed to look at the vCenter and I did not see the VM anywhere. I thought to myself now how can this be? I am not 100 % of what exactly happened for sure but I have a theory that I intend to try and duplicate at a later date.
So here is goes:
The entire environment went offline during a network issue that caused the iSCSI to drop out. We have had this issue before and after checking the NAS we discovered that the NAS is okay we initiated a reboot of the virtual hosts. Once they came back online we found that the storage adapters could not see the NAS. So from here we rebooted the NAS and we were then able to do a rescan and the LUNs all came back without issue.
Now because we have HA and DRS enabled on our cluster the VMs all started on their own, causing a whole new headache that I do not want to go into at this point. However during this process the one host evacuated all it’s VMs and sent it to the other host.
I am thinking that this is the point where the VM mix up occurred. the 2 virtual hosts were fighting for the VM and that contention caused it to become orphaned. To resolve this issue I removed the VM from the inventory on both machines and then the vmx.lck file disappeared and I was able to re-add the VM back into inventory though vCenter.
I hope this helps someone with their issue, sorry for the lack of pictures but if/when it happens again I will be sure to add it back in.