NetScaler as a Virtual Appliance is Now Available for ESX and ESXi

NetScaler VPX is now available for ESX and ESXi. 

http://www.citrix.com/English/ps2/products/feature.asp?contentID=1689968

Migrate Provisioning Server Devices

I was rebuilding a PVS infrastructure with a new database and ran into an issue with the migration.  I was trying to decide whether it would be faster to migrate the existing database or start from scratch and rebuild everything.  Nothing was in production yet and after a little research, migrating the existing PVS database to the new server could be tricky.  Instead, I decided to start from scratch and export my devices from the old PVS database and import them into the new.  Citrix wrote this helpful guide on how to export to a .CSV and I was able to get everything transferred over quickly without any issues. 

http://support.citrix.com/article/CTX118832

So, if you're migrating to a new database server and want to start fresh with a newly created farm, this is for you.  Take advantage of this opportunity to perform a redesign without having to recreate all of your devices.

vSphere Update 1 Breaks EMC Storage Viewer

The EMC storage viewer has always been a handy interface to peer into your VMware storage in the vcenter interface.  Due to depreciated changes in the plugin framework, this no longer works after update 1.  Rumor has it that version 2.1 of the plugin should be out any day now and this newer version should be fully compatible with Update 1. 

For more information on the EMC Storage Viewer, check out this article by Mr. Sakac:

http://virtualgeek.typepad.com/virtual_geek/2009/04/where-to-get-the-emc-storage-viewer-vcenter-plugin.html

XenDesktop 4 is now Available

XenDesktop 4 is available today with FlexCast, HDX, and more.  Check out the Citrix Blog for more information:


http://community.citrix.com/display/ocb/2009/11/16/XenDesktop+4+is+here!

EMC FAST (Fully Automated Storage Tiering)

There's a new functionality on the horizon with EMC Storage Arrays and the way that they handle their data.  The standard way to provision storage has always been to size your performance requirements correctly and statically assign a RAID Group with a certain number of disks running either SATA or Fibre Channel, and most recently EFD (Electronic Flash Drives).

With FAST, the goal is basically to allow EMC customers to get the best value out of their storage.  This works by letting the array automatically provision and re-provision data to the most appropriate storage tier based on  performance requirements.

Take a look at this video to see how the performance increases using FAST and EFDs.  The example is using Symmetrix, however, note that it mentions that their medium class (CLARiiON) and NAS class (Celerra) arrays will be supported as well.

http://www.youtube.com/watch?v=ofKBDvo8214

Issue Provisioning Disks with the XenDesktop Setup Wizard

I had an issue when running the XenDesktop Setup Wizard where the wizard wouldn't connect to the provisioning server.  This is because many of the ports have changed recently due to port collision issues.  To resolved this it is necessary to edit the SetupToolApplication.exe.config file and change the following settings. 


The default installation path is C:\Program Files\Citrix\XenDesktop Setup Wizard.
 Create a backup copy of this file.
 Within the section of the file:

a. Replace:
endpoint address="http://localhost:8000/pvs/mapi/commandset"
with:
endpoint address=http://localhost:54321/pvs/mapi/commandset 

b. Replace:
endpoint address="net.tcp://localhost:8001/pvs/mapi/commandset"
with:
endpoint address="net.tcp://localhost:54322/pvs/mapi/commandset"

You could read further about fixing the issue in this Citrix Support Document


http://support.citrix.com/article/CTX121658

Citrix Updates

Some new updates out recently for Xenapp and Provisioning Server


Hotfix Rollup Pack 5 for XenApp 5.0 on Server 2003

http://support.citrix.com/product/xa/v5.0_2003/

Citrix Provisioning Server 5.1 SP1

http://support.citrix.com/article/CTX122799

TCP/IP Stack Corruption on VMware Host.

I encountered an issue this week where multiple VMs on a certain ESX host start having corruption of their TCP/IP stack. The 2008 VMs were unable to retain their gateway while the 2003 VMs were unable to retain any IP info whatsoever.  While still researching the cause, there is a solution. To repair the TCP/IP stack, just run this command which comes from this KB from Microsoft.

http://support.microsoft.com/kb/299357

netsh int ip reset c:\resetlog.txt

Here is a VMware Communities post with similar symptoms.

http://communities.vmware.com/message/1336502#1336502

This thread discusses the problem as well but doesn't really mention VMware.

http://social.technet.microsoft.com/Forums/en-US/itprovistasp/thread/27fd86ad-caad-4698-9032-63550695ee3d

Some Thoughts on the New FLARE 29

I recently implemented a new CX4 with the latest version of Navisphere, FLARE 29.  There's been a lot of hype around this release because 29 now gives storage admins the ability to peer into VMware environments and even see which guest VMs exist on which LUNs. ESX hosts icons appear differently and are easy to decipher form other hosts. I really like the integration and setup was straight forward.  Just a quick wizard to run through that requires authentication to your virtual center server and your good to go.  I've only tested the new FLARE in with vsphere so I'm not sure if the interface would give similar results and visibility into a VI3.5 environment. 

Another change in FLARE 29 that appears simple at first but, in my opinion, is quite a great enhancement is an upgrade to the "connectivity status" section of Navisphere.  Host wwn's and iqn's are now grouped together in a tree structure under the hostname.  No more sorting and trying to figure out if all your initiators registered successfully or not!  What a time save if you're implementing several new hosts in an environment. 

CAUTION:  It does appear that there is a bug with FLARE 29 and Site Recovery Manager. If using SRM with MirrorView, hold off until the next version of SRM is released and do the SRM upgrade first.  Hopefully, this will also be the time a vSphere compatible version of SRM is released as well.  For more information about this issue, check out the end of this fine article from Chad Sakac.

http://virtualgeek.typepad.com/virtual_geek/2009/09/a-couple-important-alua-and-srm-notes.html

Guest Memory Utilization with Nehalem, ESX 4 and vSphere

One of the hot things to buy now with a new vSphere implementation are a set of beefy hosts with the new Nehalem processor line. This is great but there is a bug you need to watch out for. Memory usage for guests are showing 80%-90% utilized when actually there will only be 30%-50% consumed within the guest VM. This can really throw you for a loop and after some research, it appears to have something to do with Transparent Page Sharing or TPS and the way it works with MMU enabled.

You can learn more about MMU and VMware here

http://blogs.vmware.com/performance/2009/03/index.html

According to a VMware communities thread I was reading, there should be a patch coming out mid to late September. In the meantime you could disable large page sharing as a workaround, however, this will cause a big loss in performance. Check out this community thread for more info as well.

http://communities.vmware.com/thread/211585;jsessionid=C282B0ADD49F2A96E3CA99A7951DC974?start=0&tstart=0

VMWare Workstation NAT problems in Windows 7

I learned from a co-worker this weekend that they were having trouble running VMWare Workstation VMs in NAT mode on the new release of Windows 7. It appears that this is fixed in the latest beta version of VMWare Workstation but until it reaches full release there seems to be a work around. I haven't tried this yet but I've read a lot of positive responses.


1. Run the Virtual Network Editor as Administrator (Use RunAS or right click and Run as Administrator on vmnetcfg.exe)
2. Goto Host Virtual Adapters and remove all VMNet instances (VMNet1 and VMNet8 typically).
3. Click Apply.
4. Add New and Assign it the new adapter to VMnet1.
5. Click Apply.
6. Select the Host Virtual Networking tab.
7. Click the > next to VMnet1 and change the address and subnet to the ICS network (192.168.37.0 / 255.255.255.0)
8. Click Apply.
9. Go to the NAT tab and select VMNet1.
10. Click Edit and change the Gateway to the ICS gateway IP (192.168.37.1)
11. Click Apply and restart the NAT service. (Counter-intuitive, I know.)
12. Go to the DHCP tab.
13. Add VMNet1 and remove all others.
14. Click Apply.
15. Select Properties of VMNet1 in DHCP Tab.
16. Enter a Start and End Address for DHCP Scope (192.168.37.50 to 192.168.37.75 as an example)
17. Adjust client lease to a few days to avoid potential DHCP client renewal timeouts.
18. Click OK / Click Apply.
19. Select Host Only Networking for every VM that needs NAT out to your host network/internet.
20. Enable ICS (Internet Connection Sharing) on the W7 Host network card that provides connectivity. Select VMNet1 as the network card that needs access.


This thread explains the issue in more detail:

http://communities.vmware.com/message/1309197

Some more info on VMware Data Recovery

There's been a lot of talk about using VMware Data Recovery with ESX 4 and vSphere and what it does and doesn't do. We'll get the negatives out of the way first. vDR isn't a VCB replacement so if you still need a VCB proxy in your environment that will still need to be maintained. vDR doesn't write directly to tape. vDR's de-dupe technology is target based.. Therefore, you will experience disk space savings but all those redundant blocks will still be sent over the network.

Okay, now that we have some of the obvious negatives out of the way, let's talk about why vDR is really awesome. It comes as a virtual appliance so there's nothing to build or configure as far as the device goes. There's also a plugin that install directly into vSphere and your backups can be managed by the vSphere client (no 3rd party apps required!). There are many more cool features and there's also an experimental plugin that will allow you to perform file level restores from windows VMs.

vDR vs. Veeam and detailed install instructions

Here is a link that lists a vdr and veeam comparison as well as install instructions with screenshots:

http://searchstorage.techtarget.com.au/articles/34373-How-to-install-and-operate-VMware-Data-Recovery-Manager

Install and Configure Video

Here is the first part of an installation video. This is pretty cool and shows the different interfaces.

http://www.dabcc.com/media.aspx?id=505

vCenter Lifecycle Manager

OK, so this topic has already been covered in the Varrow Blogs, mostly on Calfo's Blog. I want to bring it up again, however, because I keep seeing environments that stand to benefit greatly by putting this or a similar technology into place in their ESX environment.

There's no argument that VMware makes things easier for your IT Department but could it be argued that it makes things a little too easy? Now that you can have a new windows or linux guest up, running, and configured in minutes rather than hours everyone wants part of the action, right???

Of course they do and why would you blame them? Most administrators try manage provisioning tasks manually and quickly get overwhelmed. Ed, the exchange admin, had you create him a test server last month. Is Ed done testing now? Can you shutdown Ed's VM and free up some much needed resources? We can ask Ed, but oh yeah.. that's right, Ed's on vacation for a couple of weeks. Will we forget to ask him when he gets back?

You get the picture. It can get really nasty trying to keep up with everyone and every project that requires use of your ESX infrastructure. For this reason and after some recommendations from coworkers I started looking into VMware's vCenter Lifecycle Manager. It appears that it will give IT the ability to automate much of the lifecycle process and hooks right into vCenter 2.x. I'm hoping to evaluate it in depth soon but you can check it out on VMware's page here:

https://www.vmware.com/products/lcm/

And check out Calfo's demo video he posted at the beginning of the year.

http://calfo.wordpress.com/2009/01/11/vmware-lifecycle-manager-demo-video/

Unregistered hosts with CLARiiON iSCSI

I ran into an issue recently where I was unable to register w2k8 hosts in Navisphere completely with the naviagent. The host would communicate with Navisphere and autoregister the iqns but the host itself would show a "U" (unregistered) status. Also, Navisphere would now show host info such as drive mappings etc... I spent a lot of time going over settings and rechecking configurations. It was really bizarre and an issue I had never encountered in the Fiber Channel environment. After working with EMC support, the solution finally came out. When using iSCSI on Windows servers to connect to a CLARiiON iSCSI storage system, the iSCSI NICs on the hosts cannot be the first bound NIC.

This solution is elaborated upon in the EMC solution emc191748.

You can check the binding order in a number of ways.

Use the netsh interface

1. Go to "Network and Dial-up Connections." (For Windows 2008, select "Manage Network Connections.")
2. From the toolbar select Advanced/Advanced Settings.
3. In the "Adapters and Bindings" tab, ensure that the NIC used for normal, non-iSCSI traffic is at the top of the list, followed by the iSCSI NICs.
4. If you need to change this order, a reboot is required or you can use the follow two commands to turn off/on each NIC.
   
   To disable:
   
   netsh interface set interface <interface name> DISABLED
   
   To re-enable:
   
   netsh interface set interface <interface name> ENABLED
   
   Run these two commands for each NIC.

Use the ipconfig/all command

You can use the ipconfig/all command from a command prompt. For Windows 2000 and 2003, the NICs will display in reverse order, that is, the first NIC listed is the lowest NIC in the binding order. For Windows 2008, the order of the NICs will follow the correct sequence, that is, the first NIC listed will be the NIC bound.

Using netstat-rn command

For Windows 2008 servers you can use the netstat-rn command.

The numbers listed in the left column reflect the binding order with the lowest number being the first NIC bound. For Windows 2000 and 2003, it is the opposite.

C:\Users\Administrator>netstat -rn
===================================================================
Interface List
10 ...00 14 22 b1 7b ae .........Intel(R) PRO/1000 MB Dual Port Server Connection* (See note below.)
11 ...00 14 22 b1 7b af ...... ...Intel(R) PRO/1000 MB Dual Port Server Connection #2** (See note below.)

1 ................................... Software Loopback Interface 1*** (See note below.)

12 ...02 00 54 55 4e 01 ......... Teredo Tunneling Pseudo-Interface
13 ...00 00 00 00 00 00 00 e0 isatap.{14388A07-03E6-48AE-A713-D835413A72A5}
14 ...00 00 00 00 00 00 00 e0 6TO4 Adapter
16 ...00 00 00 00 00 00 00 e0 isatap.{6838E21C-4151-41EB-89E6-7C005E8E58A2}

* Second bound NIC. This is the first, real NIC. It does show up in the GUI.

** Third bound NIC. This is the second, real NIC. It does show up in the GUI.

*** First bound NIC. This is the localhost and will not show up in the GUI list above.

Note See solution emc159428.

Adding a Backup URL into Citrix Web Interface

Creating a backup URL is one of several strategies that exist for designing a redundant Web Interface in your XenApp farm if you are using the XA plugin for hosted apps (PN Agent). There is always the option of creating multiple Web Interface servers and using load balancing appliances. There is also the option of using a round robin approach with DNS (poor man's load balancing). However, using appliances may be "over the top" in an environment without heavy utilization and the DNS approach isn't intelligent enough to detect a failed server and would require manual removal of the failed server entry.

Adding a backup URL to Web Interface for your hosted plugin allows a seamless failover to a second site. The process starts by specifying the backup URL into the Web Interface configuration. After that, the URL is pushed out to all of the hosted apps plugins upon their next successful connection. In the event that your primary Web Interface fails, the hosted apps plugin will detect the failure and automatically attempt to connect with backup web interface server specified in the backup URL configuration. Instructions on how to add the backup site can be found here:

http://support.citrix.com/proddocs/index.jsp?topic=/web-interface/wi-specify-backup-urls.html

Enhanced VMotion

EVC (Enhanced VMotion Compatibility) is a feature in ESX Clusters that allows you to VMotion among different processors of the same family. To enable this feature, you need to start with a new cluster and add your hosts in after verifying they have the correct settings turned on in the BIOS.

It's common practice to turn Virtualization on in the advanced CPU settings before building a new host. On some hosts, such as HP, the "No Execute" (also referred to as NX or XD) bit needs to be enabled in the BIOS as well.

There's a lot of potential for using EVC between hosts of the same processor family. Following are some great links with more detail about EVC, its use cases, and some intel compatibility specifications.

http://www.itworld.com/virtualization/56292/understanding-vmware-evc

http://www.vmguru.nl/wordpress/2009/06/vmware-evc-cluster-what-is-that/

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1991

Install issues combining Citrix Web Interface and other products on x64.

Often times when setting up Citrix environments outside of production, it makes sense to combine multiple pieces onto one server. After recently setting up XenDesktop using an x64 environment, there was an unexpected issue trying to combine the XenDesktop DDC, Web Interface, and the Licensing Management Console. This combination has never been a problem in a 32-bit environment so what could be different in x64? The answer... IIS

It was all very confusing at first because the License Server installation happened at the very beginning and worked without a hitch. Licenses were added and then DDS was installed on the same machine. The complete desktop environment was built without encountering any problems until time to configure Web Interface. At that point, no web pages on that machine were accessible, not even the Licensing Management Console which had worked fine previously. After some research, the cause of the problem was found along with the solution midway through the release notes for Presentation Server 4.5 with Feature Pack 1 which can be found here:

http://support.citrix.com/article/CTX112222

The DDC behaves a lot like XenApp and the same information regarding x64 applies. Quoting from the release notes found above, Citrix points out that "On a 64-bit version of Windows, the Citrix Web Interface installation will enable 32-bit Web extension support in IIS and this will disable 64-bit extension support."

And the solution.... Install Web Interface FIRST.


Still quoting from the above article "Due to this requirement, if you are installing the Web Interface on a 64-bit version of the Windows operating system, ensure that you install Web Interface prior to installing any other Citrix software, including Citrix Presentation Server and Citrix License Management Console. This particular order of installation allows the products to adapt to the 32-bit support in IIS. If you install these products in an incorrect order, the Web server may produce errors messages when it is accessed, such as “Service unavailable.”

So, uninstall and reinstall making sure to do Web Interface first. This will prep IIS for all the subsequent pieces.

Deleting Stub VMs and their Meta Data in XenServer

Sometimes on a failed XenConvert, the process will leave "stubbed" VM meta data behind on the XenServer. Any attempts to delete this VM through Xen Center will fail. Even though the Xen Convert process will have to be re-evaluated and restarted, there's a quick two step process to remove the stubbed VM from XenServer.

1. From the console of the XenServer containing the VM get a list of uuids by running "xe vm-list".

2. Copy the uuid of the VM in question and then run this command to destroy it "xe vm-destroy uuid="

It might take a couple of minutes for this command to complete but when it does the reference to the VM is now also removed from XenCenter. If a disk was associated with it, there should now be an option to delete it or it can also be attached to a different VM. It's possible that the data on this disk is salvageable but should be verified that it is in tact and consistent.

XenDesktop Receiver Embedded Client (Full-Screen-Only)

I had an opportunity last Friday to experiment with the Desktop Receiver Embedded Client for XenDesktop. After my past experiences with Thin Clients, I knew the embedded client would make a lot of sense for both administration and ease of use. The install package is fairly small (11MB) and the functionality is almost seamless.

After logging into a standard MS Windows GINA, the user will be redirected to a full screen only edition of their XenDesktop. By enabling write filters on the thin clients and also having VMs reboot after a XenDesktop user log off, laborious desktop management becomes almost non-existent. The user experience takes place in an almost completely read-only environment that moves with them wherever they go.

According to the Desktop Receiver Admin Guide, the embedded client replaces the shell for each Windows user except for the user account that installs the software. Therefore, any future windows user that logs in and is successfully authenticated will also automatically be given a full-screen-only desktop. Admin accounts, such as members of Domain Admins, are not excluded from a full-screen-only desktop. Any changes that need to be made to the client device will require the account that was used to install the embedded software.

It seems that the software presents the full-screen-only desktop by modifying the "Shell" registry key in: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon from "explorer.exe" to "pnagent.exe".

Citrix also mentions that if the account used to install DesktopApplianceLock_en.msi is later deleted, you will not be able to log on and administer the endpoint device.

Dell ExtPart

I learned about this great tool from a colleague of mine this week. Dell ExtPart conveniently extends any windows partition including your c: drive! Even more amazing is this tool has been around since 2003 and I'm just now hearing about it... What's up with that!?

Working in IT and especially with VMWare, there are always times where you add more disk or a case where you forgot to format properly during OS install and you need to expand your system disk. In the past, I've always had to re-install or turn to tools such as partition magic or opensource software for physical machines. VMs were somewhat easier because you could mount the system disk on another VM perform the expansion.

Now the process couldn't be easier. Download this 36KB utility at the link below, extract and run extpart.exe, follow the promtps, and in 15-30 seconds you're done!

http://support.dell.com/support/downloads/download.aspx?c=us&cs=19&l=en&s=dhs&releaseid=R64398&formatcnt=2&fileid=83929

NOTE: A word of caution. When I first ran the utility and it asked me for an expand size, I entered the total size that I wanted my system drive to be. This is wrong because it is actually asking how much space you would like to add to your existing drive size. If you want to go from 15G to 20G, then you would only enter 5G into the prompt.