TCP/IP Stack Corruption on VMware Host.

I encountered an issue this week where multiple VMs on a certain ESX host start having corruption of their TCP/IP stack. The 2008 VMs were unable to retain their gateway while the 2003 VMs were unable to retain any IP info whatsoever.  While still researching the cause, there is a solution. To repair the TCP/IP stack, just run this command which comes from this KB from Microsoft.

http://support.microsoft.com/kb/299357

netsh int ip reset c:\resetlog.txt

Here is a VMware Communities post with similar symptoms.

http://communities.vmware.com/message/1336502#1336502

This thread discusses the problem as well but doesn't really mention VMware.

http://social.technet.microsoft.com/Forums/en-US/itprovistasp/thread/27fd86ad-caad-4698-9032-63550695ee3d

Some Thoughts on the New FLARE 29

I recently implemented a new CX4 with the latest version of Navisphere, FLARE 29.  There's been a lot of hype around this release because 29 now gives storage admins the ability to peer into VMware environments and even see which guest VMs exist on which LUNs. ESX hosts icons appear differently and are easy to decipher form other hosts. I really like the integration and setup was straight forward.  Just a quick wizard to run through that requires authentication to your virtual center server and your good to go.  I've only tested the new FLARE in with vsphere so I'm not sure if the interface would give similar results and visibility into a VI3.5 environment. 

Another change in FLARE 29 that appears simple at first but, in my opinion, is quite a great enhancement is an upgrade to the "connectivity status" section of Navisphere.  Host wwn's and iqn's are now grouped together in a tree structure under the hostname.  No more sorting and trying to figure out if all your initiators registered successfully or not!  What a time save if you're implementing several new hosts in an environment. 

CAUTION:  It does appear that there is a bug with FLARE 29 and Site Recovery Manager. If using SRM with MirrorView, hold off until the next version of SRM is released and do the SRM upgrade first.  Hopefully, this will also be the time a vSphere compatible version of SRM is released as well.  For more information about this issue, check out the end of this fine article from Chad Sakac.

http://virtualgeek.typepad.com/virtual_geek/2009/09/a-couple-important-alua-and-srm-notes.html

Guest Memory Utilization with Nehalem, ESX 4 and vSphere

One of the hot things to buy now with a new vSphere implementation are a set of beefy hosts with the new Nehalem processor line. This is great but there is a bug you need to watch out for. Memory usage for guests are showing 80%-90% utilized when actually there will only be 30%-50% consumed within the guest VM. This can really throw you for a loop and after some research, it appears to have something to do with Transparent Page Sharing or TPS and the way it works with MMU enabled.

You can learn more about MMU and VMware here

http://blogs.vmware.com/performance/2009/03/index.html

According to a VMware communities thread I was reading, there should be a patch coming out mid to late September. In the meantime you could disable large page sharing as a workaround, however, this will cause a big loss in performance. Check out this community thread for more info as well.

http://communities.vmware.com/thread/211585;jsessionid=C282B0ADD49F2A96E3CA99A7951DC974?start=0&tstart=0

VMWare Workstation NAT problems in Windows 7

I learned from a co-worker this weekend that they were having trouble running VMWare Workstation VMs in NAT mode on the new release of Windows 7. It appears that this is fixed in the latest beta version of VMWare Workstation but until it reaches full release there seems to be a work around. I haven't tried this yet but I've read a lot of positive responses.


1. Run the Virtual Network Editor as Administrator (Use RunAS or right click and Run as Administrator on vmnetcfg.exe)
2. Goto Host Virtual Adapters and remove all VMNet instances (VMNet1 and VMNet8 typically).
3. Click Apply.
4. Add New and Assign it the new adapter to VMnet1.
5. Click Apply.
6. Select the Host Virtual Networking tab.
7. Click the > next to VMnet1 and change the address and subnet to the ICS network (192.168.37.0 / 255.255.255.0)
8. Click Apply.
9. Go to the NAT tab and select VMNet1.
10. Click Edit and change the Gateway to the ICS gateway IP (192.168.37.1)
11. Click Apply and restart the NAT service. (Counter-intuitive, I know.)
12. Go to the DHCP tab.
13. Add VMNet1 and remove all others.
14. Click Apply.
15. Select Properties of VMNet1 in DHCP Tab.
16. Enter a Start and End Address for DHCP Scope (192.168.37.50 to 192.168.37.75 as an example)
17. Adjust client lease to a few days to avoid potential DHCP client renewal timeouts.
18. Click OK / Click Apply.
19. Select Host Only Networking for every VM that needs NAT out to your host network/internet.
20. Enable ICS (Internet Connection Sharing) on the W7 Host network card that provides connectivity. Select VMNet1 as the network card that needs access.


This thread explains the issue in more detail:

http://communities.vmware.com/message/1309197

Some more info on VMware Data Recovery

There's been a lot of talk about using VMware Data Recovery with ESX 4 and vSphere and what it does and doesn't do. We'll get the negatives out of the way first. vDR isn't a VCB replacement so if you still need a VCB proxy in your environment that will still need to be maintained. vDR doesn't write directly to tape. vDR's de-dupe technology is target based.. Therefore, you will experience disk space savings but all those redundant blocks will still be sent over the network.

Okay, now that we have some of the obvious negatives out of the way, let's talk about why vDR is really awesome. It comes as a virtual appliance so there's nothing to build or configure as far as the device goes. There's also a plugin that install directly into vSphere and your backups can be managed by the vSphere client (no 3rd party apps required!). There are many more cool features and there's also an experimental plugin that will allow you to perform file level restores from windows VMs.

vDR vs. Veeam and detailed install instructions

Here is a link that lists a vdr and veeam comparison as well as install instructions with screenshots:

http://searchstorage.techtarget.com.au/articles/34373-How-to-install-and-operate-VMware-Data-Recovery-Manager

Install and Configure Video

Here is the first part of an installation video. This is pretty cool and shows the different interfaces.

http://www.dabcc.com/media.aspx?id=505

vCenter Lifecycle Manager

OK, so this topic has already been covered in the Varrow Blogs, mostly on Calfo's Blog. I want to bring it up again, however, because I keep seeing environments that stand to benefit greatly by putting this or a similar technology into place in their ESX environment.

There's no argument that VMware makes things easier for your IT Department but could it be argued that it makes things a little too easy? Now that you can have a new windows or linux guest up, running, and configured in minutes rather than hours everyone wants part of the action, right???

Of course they do and why would you blame them? Most administrators try manage provisioning tasks manually and quickly get overwhelmed. Ed, the exchange admin, had you create him a test server last month. Is Ed done testing now? Can you shutdown Ed's VM and free up some much needed resources? We can ask Ed, but oh yeah.. that's right, Ed's on vacation for a couple of weeks. Will we forget to ask him when he gets back?

You get the picture. It can get really nasty trying to keep up with everyone and every project that requires use of your ESX infrastructure. For this reason and after some recommendations from coworkers I started looking into VMware's vCenter Lifecycle Manager. It appears that it will give IT the ability to automate much of the lifecycle process and hooks right into vCenter 2.x. I'm hoping to evaluate it in depth soon but you can check it out on VMware's page here:

https://www.vmware.com/products/lcm/

And check out Calfo's demo video he posted at the beginning of the year.

http://calfo.wordpress.com/2009/01/11/vmware-lifecycle-manager-demo-video/

Unregistered hosts with CLARiiON iSCSI

I ran into an issue recently where I was unable to register w2k8 hosts in Navisphere completely with the naviagent. The host would communicate with Navisphere and autoregister the iqns but the host itself would show a "U" (unregistered) status. Also, Navisphere would now show host info such as drive mappings etc... I spent a lot of time going over settings and rechecking configurations. It was really bizarre and an issue I had never encountered in the Fiber Channel environment. After working with EMC support, the solution finally came out. When using iSCSI on Windows servers to connect to a CLARiiON iSCSI storage system, the iSCSI NICs on the hosts cannot be the first bound NIC.

This solution is elaborated upon in the EMC solution emc191748.

You can check the binding order in a number of ways.

Use the netsh interface

1. Go to "Network and Dial-up Connections." (For Windows 2008, select "Manage Network Connections.")
2. From the toolbar select Advanced/Advanced Settings.
3. In the "Adapters and Bindings" tab, ensure that the NIC used for normal, non-iSCSI traffic is at the top of the list, followed by the iSCSI NICs.
4. If you need to change this order, a reboot is required or you can use the follow two commands to turn off/on each NIC.
   
   To disable:
   
   netsh interface set interface <interface name> DISABLED
   
   To re-enable:
   
   netsh interface set interface <interface name> ENABLED
   
   Run these two commands for each NIC.

Use the ipconfig/all command

You can use the ipconfig/all command from a command prompt. For Windows 2000 and 2003, the NICs will display in reverse order, that is, the first NIC listed is the lowest NIC in the binding order. For Windows 2008, the order of the NICs will follow the correct sequence, that is, the first NIC listed will be the NIC bound.

Using netstat-rn command

For Windows 2008 servers you can use the netstat-rn command.

The numbers listed in the left column reflect the binding order with the lowest number being the first NIC bound. For Windows 2000 and 2003, it is the opposite.

C:\Users\Administrator>netstat -rn
===================================================================
Interface List
10 ...00 14 22 b1 7b ae .........Intel(R) PRO/1000 MB Dual Port Server Connection* (See note below.)
11 ...00 14 22 b1 7b af ...... ...Intel(R) PRO/1000 MB Dual Port Server Connection #2** (See note below.)

1 ................................... Software Loopback Interface 1*** (See note below.)

12 ...02 00 54 55 4e 01 ......... Teredo Tunneling Pseudo-Interface
13 ...00 00 00 00 00 00 00 e0 isatap.{14388A07-03E6-48AE-A713-D835413A72A5}
14 ...00 00 00 00 00 00 00 e0 6TO4 Adapter
16 ...00 00 00 00 00 00 00 e0 isatap.{6838E21C-4151-41EB-89E6-7C005E8E58A2}

* Second bound NIC. This is the first, real NIC. It does show up in the GUI.

** Third bound NIC. This is the second, real NIC. It does show up in the GUI.

*** First bound NIC. This is the localhost and will not show up in the GUI list above.

Note See solution emc159428.