Chronophage.net Blog

So I put a hypervisor in your hypervisor, so you can virtualize while you virtualize.

Ubuntu Running VirtualBox, running Windows 7, running vSphere Client connected to vSphere hosted on ESXi running two ESX hosts and Nexenta, together running an Ubuntu Netbook VM on shared storage.

This shows the storage, both hosted in the lab enviroment (nexenta, sexy) and external (datastore(1) CALLY – an NFS server)

Requirements:
VMware vSphere, lots of RAM, a decent amount of disk space, a fairly recent copy of 64bit Windows (I used Server 2008 R2) ESX and vSphere Server iso and exe files. Iron will. Patience. Some sort of NAS distribution (I used FreeNAS.)

So I wanted to set up vSphere. We use it quite a bit at work, but I wanted to try it in a full, yet simulated environment. I have a desktop that I bought in ’07 for gaming. Although it’s a bit long in the tooth, it had enough CPU (e6600) to get the job done. Also, it was before Intel released core2duo chips that lacked VT extensions. I added 4GB of RAM for a total of 6GB. I had installed Ubuntu on it, as I lacked an extra Windows license, and frankly, it’s good enough.

I installing a demo copy of VMware Workstation, as I like Fusion at work. Workstation is a seamless interface that allows juggling multiple VMs with an intuitive network set up and centralized management. I may buy it.

I first set up two ESX hosts. A very painless and quick install. Then I installed Windows 2008R2. Installing vSphere on it took a bit of time, but was relatively straight forward. I didn’t feel like futzing with SQL server, so I installed SQL Express via the default setup. Again, very straightforward. If this were more of an active lab, or production environment, I would’ve installed SQL server, as SQL Express can’t hold more than a few hosts’ worth of data. vSphere generates a TON of statistical and historical information.

After adding the hosts, and putzing around a bit, I installed VMware Update Manager, and VMware Converter. I’ll play with VUM, but probably not Converter.

To get vMotion to work, I had to install a VMKernel port. iSCSI and NAS required a Service console port. vMotion had to be enabled on the port, and an iSCSI software hba need to be enabled.

After getting the front end systems up, I installed FreeNAS in another VM in Workstation. I assigned it in 2 80GB, thinly provisioned disks for NAS exports, and 10 8GB disks for various other applications. In FreeNAS, one can add disks on the fly. After figuring out that FreeNAS did *everything* in a hierarchical manner, setting up NFS exports was trivial.

Back in vSphere, exposing the ESX hosts to the NFS mounts was trivial, and I utilized all the space as two 80GB datastores. The I/O hit wasn’t trivial. The lack of network segmentation, even virtually, takes its toll.

iSCSI was a bit more difficult. FreeNAS had a few more levels of configuration than I expected, so I missed some steps. Basically, you set up an Extant, and then set up a target to it, and then hit apply. The last step bit me a few times

I exposed 2 disks as unformatted block devices, and created two files on the UFS formatted file system to the other two. Setting up a software RAID is pretty easy in FreeNAS, but I was just poking around, so I didn’t care about redundancy. Either option seemed the same. Curiously, each extent showed as a separate “disk” each with LUN 0. I’m sure there’s a way to change it to one “disk” with several LUNs, but I’m playing with vSphere, not FreeNAS.

Back in vSphere, I had to enable iSCSI, point the hosts’ iSCSI HBAs to the NAS/SAN VM. I had four LUNs available: two using the “write to file” method, and two as block devices. I set up the first Datastore with the “file” LUN first, and extended it with the last block device LUN. I did the inverse with the remaining two. That gave me two 16GB Datastores. Again, very easy.

FreeNAS gave me trouble with iSCSI and ZFS. I was able to set up a ZFS/RAID 5 pool, but the native method to attach it to iSCSI didn’t seem to work. I’m sure I’m just missing something, the the volume wasn’t available to attach to an iSCSI extent. Maybe I’ll figure it out, maybe I won’t. Again, I’m more interested in vSphere, and not FreeNAS.

Seeing as I was running my virtualization environment in an virtualized environment, I didn’t have VT extensions available to my ESX hosts. That means that they could only run 32bit VMs on them. Fine for testing, but I only had 64bit ISOs. So a quick download of Ubuntu Desktop Edition, 32Bit, and I had an Ubuntu VM running in an ESX VM running on Ubuntu A surreal experience. I don’t think VUM updates Ubuntu, so I’ll have to dig around for an old XP iso. I’m looking forward to playing with VUM, and Distributed Switches.

Things I would do differently:

OpenFiler, which I tried first, did not work intuitively. None of the HOW TOs I found online matched the interface I was looking at, and the instructions cost 40 quid (is that how you say that, I am very not British) . No thanks.

If I had more resources, and time, I would set up things more like what I would expect to see in an enterprise environment. More switches, better segmentation, more redundancy in the storage, etc. My lab would get an F in Infrastructure Design. As it is though, the VM within a VM is *painfully* slow.

vSphere is fun, and I’d love to learn more. Soon, due to Workstation’s demo license expiring, I will have to dismantle my lab. Perhaps I can recreate it with KVM and virt-manager (ouch!)

Updates may follow.

Hey, it works! I’m moving a file (on a GIG-E switch between VLANs) at around 3MBps.

Thanks to http://linux-bsd-sharing.blogspot.com/2008/09/howto-setup-nfs-server-on-freebsd.html

I bought a gaming laptop. I bought it thinking that I could do a lot with the nice, powerful chip that came with it. My first disappointment was the horribly outdated NVIDIA drivers that came with the laptop. I could not play WoW, at all. I found this odd for a gaming laptop. I could forgive them for giving me 4gigs of Ram, but a 32-bit operating system. My second is this, I cannot enable VT extensions on my CPU, even though it clearly supports it. I need to be able to run VMWare and/or VirtualPC for various work projects. This is highly disappointing. Ultimately, my prime gaming laptop may end up collecting dust in the closet, a constant reminder of why no one should buy Gateway.

One of Chronophage.net’s servers is virtualized inside of a Dell 1500. It runs like a champ. I think this is the future of hosting, especially with power and cooling costs going up. I should work on my VTSP now.

Ares.chronophage.net used to be my experimental FreeBSD box. It ran on a T2100 dual-core consumer system with 1.5 gigs of ram and an 80GB harddrive (a freebie that Dell sent to one of my company’s customers, ESATA) The system was slow, unreliable, and prone to crashing. With VMWare, things are scaling really well. The tools were in the ports tree, and it was painless to install. The system converter worked better than expected (since FreeBSD is not officially supported) All in all, this is a successful experiment.