At some stage while in contact with VMware support, they may give you drivers to try. These drivers will be in VIB (vSphere Installation Bundle) format and will look something like vmware-esx-drivers-net-bnx2x-400.1.52.12.v40.4-1.0.4.00000.x86_64.vib. I don’t know why this is not documented anywhere in the official docs, but here’s how to install these driver bundles into ESXi.

1. scp the file to the ESXi server
2. Run “esxupdate -b <filename>.vib –nosigcheck –nodeps update”.
3. Run “esxupdate query –vib-view” to confirm that the driver bundle is installed. You may need to reboot for the driver to take effect.

If you are fiddling with different driver revisions, you may need to remove a later version if you want to rollback to a previous version, otherwise ESXi won’t let you reinstall an older bundle. To remove a driver bundle use the following command:

esxupdate -b <driver bundle name> remove

You can get the driver bundle name by running “esxupdate query –vib-view”.

When you forcefully remove a Datastore while ESXi servers are still connected to it, all sorts of weird and wonderful things can happen. We recently had a team member accidentally install ESXi onto a LUN, blowing away the Datastore that was on it. ESXi servers become unstable, the VM’s that were running from the Datastore go into a zombie state where they may respond to pings but are not fully there because they are still running from memory but the disks have been removed from under them. The vCenter server will exhibit high DB load as the ESXi servers try to update the statuses of VM’s which aren’t actually there anymore. It’s a mess and we’ve recently had to go through this. It took us an entire day to discover what really happened because the person that did it didn’t even realise he had done it, so we had to check LUN presentation and the rest of it. It had actually stumped us making us think the VMFS filesystem had somehow gotten corrupted, until VMware support jumped onto our servers through WebEx and found the problem.

Here’s how you clean up the mess.

1. Turn off HA and DRS at the cluster level because it gets in the way
2. Remove any greyed out VM’s from vCenter Server.
3. Log into each ESXi server individually and remove the “Unknown” greyed out VM’s from Inventory.
4. SSH into each server at a time, /sbin/services.sh stop, cd /opt/vmware/vmware/uninstallers and uninstall both aam and vpxa. /sbin/services.sh start. Add back into cluster/VC, doing this will resync the ESX server’s inventory with vCenter Server.
5. Reboot each ESX server in turn to remove all zombie processes/VMs which will still be running in the background.
6. Now we need to clean up references to the old Datastore. Go into Inventory -> Datastores, if you still see the old datastore there, click on it, then click on the Virtual Machines tab, then remove anything from there from Inventory. Since none of these VMs exist anymore, it’s a safe operation. The datastore should disappear after that.

It’s not a pretty process. If you can, disconnect any fibre cables before the install, but in the case of blade servers, just be very very careful.

We’ve been having this weird problem for a while where if sitting idle for a day or two, a VM would disconnect from the network and otherwise be unavailable until after a few minutes if you try to vMotion it. But after you reboot it, it worked fine. Networking checked out. Storage checked out. What could be wrong??

Turned out the culprit was memory overcommitment on our ESX servers. Who would have thought! We had more memory allocated to VMs than there was physical memory, even though the green memory indicator said otherwise.

A bit of background as to why this happens. ESX uses a number of techniques to conserve memory which allow you to use physical memory efficiently. When memory is starting to get scarce, ESX will engage a “balloon driver” in idle VMs with VMware Tools installed. This driver will inflate causing apps in memory to write out to the operating system swap. ESX will then reclaim this memory for use in other VMs.

If memory is that tight that it can’t make the OS swap and it can’t grab memory from elsewhere, as a last resort it will start to use VM Swap (different of guest OS swap). When ESX starts a VM, it writes a file of the exact same size as memory is allocated to it. ESX will then swap the more idle VMs to this swap file without the guest knowing about it. This has a major impact on performance and….you guessed it, vMotion. Your VM will be unavailable while ESX swaps everything back into memory. You can use the performance graphs to determine how badly your VM will be affected if it has been swapped. I’ve seen a few megs worth of swap not affect a VM. But a few hundred megs will.

The best way to make ESX avoid having to use VM swap is to have enough physical memory (duh!). Failing that, you can ‘reserve’ memory for the more important VMs using memory reservations. If you set a VMs memory reservation to the amount of memory it has, ESX will never swap it. Eg if you allocate 4096MB to a VM and set it’s memory reservation to 4096MB.

Update: Aparently Update 5 resolves this problem but I haven’t tested it yet.