Chronophage.net Blog

So I put a hypervisor in your hypervisor, so you can virtualize while you virtualize.

Ubuntu Running VirtualBox, running Windows 7, running vSphere Client connected to vSphere hosted on ESXi running two ESX hosts and Nexenta, together running an Ubuntu Netbook VM on shared storage.

This shows the storage, both hosted in the lab enviroment (nexenta, sexy) and external (datastore(1) CALLY – an NFS server)

Elastic Sky X. That’s what ESX stands for. Crazy, right?

Well, more playing means more caveats

First, something I forgot to mention yesterday. To get ESX working in Ubuntu, Workstation needs to be able to put the vmnet interfaces into promiscuous mode. That requires allowing the user or group that you use to start Workstation to have read/write permissions over the vmnet devices in /dev. A simple chmod will do the trick.

Now, back to our story…

I decided to try to add some additional network paths to hopefully improve I/O. In the last post, I mentioned the VM within a VM was *painfully* slow. Well, change *painfully* to *detrimentally*. The whole environment was prone to timeouts, something that vSphere doesn’t always seem recover from gracefully.

This is understandable. Enterprise software expects an enterprise environment. Something that day hackers should appreciate. (see the perpetual “Linux Vs BSD debate” ) Dinking around is going to generate problems that one would not see in a production environment. It’s a pet peeve of mine when people review software, give it inadequate resources, and complain about the end results.

That being said, I had problems.

Adding a second nic to the ESX VMs required 2 restarts per host. Arbitrarily. The first host’s second NIC did not work via DHCP (again, not something I would rely on in production) and had to be manually configured. The second host dropped all of its routing information, and again, needed to be set up manually. I attempted a restart on both, just to see what would happen. The first host again had to be set up manually. The second came back fine or so I thought…

After playing the restart shuffle, I had both NICs in place. I decided to try creating a distributed switch. Creating a distributed switch is a three part process, that requires two views. First you create the switch in the Networking view, assigning an uplink NIC to it. Then you configure the ports. Then, back in “Hosts and Clusters” you have to add the port type (VMkernel, Service Console or VM Network) After setting it up, the distributed switch propagates to all of the hosts that you’ve assigned to it. It’s more or less a way to have a consistently configured network across multiple hosts. Very slick, but very expensive. I have a feeling that this methodology will be come more standard amongst VMware’s competitors, and will be rolled into the product at large, or at least lesser license levels. Still, networking, as my boss is prone to say, is like plumbing. If it is done correctly, there’s no need to redo it.

Unfortunately, my second ESX host failed to take the Distributed Switch configuration due to it timing out. This caused the host’s network information to become broken. How broken? Well, I poked around for a bit, and decided it was faster, and easier to install a new ESX VM. If I have time, I’ll look at the host and figure out what’s wrong, as learning to troubleshoot vSphere is a secondary goal of mine for this project. However, for now, this new ESX VM gave me an opportunity to play with the second big “Enterprise Plus” feature. Host Profiles.

Host Profiles allow you to create templates based off of existing hosts. You can then store the Profile, update it, and apply it to other hosts. After bringing the new ESX server online, I created a profile from my original ESX host. I then put the new one into Maintenance mode, attached the profile, and applied it.

After a quick summary of what changes it was going to make, It started the application process. I don’t know how it ended, as it was midnight this morning, and It was well past time to hit the hay. I assume things went well

It was interesting to see in a few lines, all that I’ve done to configure my lab machines, and it will be nice not having to do it again. Although configuring ESX is hardly an arduous chore, (well, it’s slightly painful with my lab’s limited resources) I can see how configuring more than a handful at a time would be obnoxious. Host Profiles aren’t exactly necessary as, like networking, you’re not going to constantly provision new servers. If you’re in an environment that does, you’re probably a large company that can afford the licensing fees, and the cost/benefit factor is in your favor. Still, it would be nice if you could simply copy the config from one server to another in lesser editions of vSphere, without storing the profiles or being able to update them. That would be a good compromise, and I think will become standard with VM infrastructures across the board. Red Hat does something similar with it’s OS centric Satellite/Spacewalk servers, which handle both initial configuration and update management. It’s not nearly as slick, as you do need to “script” installations.

Well, that’s day two of the “fully functioning” lab. We’ll see if the Host Profile worked on the new ESX VM

Requirements:
VMware vSphere, lots of RAM, a decent amount of disk space, a fairly recent copy of 64bit Windows (I used Server 2008 R2) ESX and vSphere Server iso and exe files. Iron will. Patience. Some sort of NAS distribution (I used FreeNAS.)

So I wanted to set up vSphere. We use it quite a bit at work, but I wanted to try it in a full, yet simulated environment. I have a desktop that I bought in ’07 for gaming. Although it’s a bit long in the tooth, it had enough CPU (e6600) to get the job done. Also, it was before Intel released core2duo chips that lacked VT extensions. I added 4GB of RAM for a total of 6GB. I had installed Ubuntu on it, as I lacked an extra Windows license, and frankly, it’s good enough.

I installing a demo copy of VMware Workstation, as I like Fusion at work. Workstation is a seamless interface that allows juggling multiple VMs with an intuitive network set up and centralized management. I may buy it.

I first set up two ESX hosts. A very painless and quick install. Then I installed Windows 2008R2. Installing vSphere on it took a bit of time, but was relatively straight forward. I didn’t feel like futzing with SQL server, so I installed SQL Express via the default setup. Again, very straightforward. If this were more of an active lab, or production environment, I would’ve installed SQL server, as SQL Express can’t hold more than a few hosts’ worth of data. vSphere generates a TON of statistical and historical information.

After adding the hosts, and putzing around a bit, I installed VMware Update Manager, and VMware Converter. I’ll play with VUM, but probably not Converter.

To get vMotion to work, I had to install a VMKernel port. iSCSI and NAS required a Service console port. vMotion had to be enabled on the port, and an iSCSI software hba need to be enabled.

After getting the front end systems up, I installed FreeNAS in another VM in Workstation. I assigned it in 2 80GB, thinly provisioned disks for NAS exports, and 10 8GB disks for various other applications. In FreeNAS, one can add disks on the fly. After figuring out that FreeNAS did *everything* in a hierarchical manner, setting up NFS exports was trivial.

Back in vSphere, exposing the ESX hosts to the NFS mounts was trivial, and I utilized all the space as two 80GB datastores. The I/O hit wasn’t trivial. The lack of network segmentation, even virtually, takes its toll.

iSCSI was a bit more difficult. FreeNAS had a few more levels of configuration than I expected, so I missed some steps. Basically, you set up an Extant, and then set up a target to it, and then hit apply. The last step bit me a few times

I exposed 2 disks as unformatted block devices, and created two files on the UFS formatted file system to the other two. Setting up a software RAID is pretty easy in FreeNAS, but I was just poking around, so I didn’t care about redundancy. Either option seemed the same. Curiously, each extent showed as a separate “disk” each with LUN 0. I’m sure there’s a way to change it to one “disk” with several LUNs, but I’m playing with vSphere, not FreeNAS.

Back in vSphere, I had to enable iSCSI, point the hosts’ iSCSI HBAs to the NAS/SAN VM. I had four LUNs available: two using the “write to file” method, and two as block devices. I set up the first Datastore with the “file” LUN first, and extended it with the last block device LUN. I did the inverse with the remaining two. That gave me two 16GB Datastores. Again, very easy.

FreeNAS gave me trouble with iSCSI and ZFS. I was able to set up a ZFS/RAID 5 pool, but the native method to attach it to iSCSI didn’t seem to work. I’m sure I’m just missing something, the the volume wasn’t available to attach to an iSCSI extent. Maybe I’ll figure it out, maybe I won’t. Again, I’m more interested in vSphere, and not FreeNAS.

Seeing as I was running my virtualization environment in an virtualized environment, I didn’t have VT extensions available to my ESX hosts. That means that they could only run 32bit VMs on them. Fine for testing, but I only had 64bit ISOs. So a quick download of Ubuntu Desktop Edition, 32Bit, and I had an Ubuntu VM running in an ESX VM running on Ubuntu A surreal experience. I don’t think VUM updates Ubuntu, so I’ll have to dig around for an old XP iso. I’m looking forward to playing with VUM, and Distributed Switches.

Things I would do differently:

OpenFiler, which I tried first, did not work intuitively. None of the HOW TOs I found online matched the interface I was looking at, and the instructions cost 40 quid (is that how you say that, I am very not British) . No thanks.

If I had more resources, and time, I would set up things more like what I would expect to see in an enterprise environment. More switches, better segmentation, more redundancy in the storage, etc. My lab would get an F in Infrastructure Design. As it is though, the VM within a VM is *painfully* slow.

vSphere is fun, and I’d love to learn more. Soon, due to Workstation’s demo license expiring, I will have to dismantle my lab. Perhaps I can recreate it with KVM and virt-manager (ouch!)

Updates may follow.

[Originally Appeared 02/04/2010 blogs.iphouse.net]

One of the many terms you’ll hear thrown around an internet service provider is Request For Comments, aka, RFC: “This isn’t per the RFC!” or “We follow the RFC!” or “Read the RFC!” So what is an RFC, and why do you want to know what it says.

RFCs are, in a nutshell, the description of how a program, or procedure should work. The history of RFC is long and boring, but basically, they’ve been around since the ARPANET Project began, as written or typed memo that were literally Requests for Comments, open ended questions that someone wanted to solicit answers to. As ARPANET grew, RFCs became the standard way to record procedure, and a way for people to implement the fundamental technologies that make up the Internet as it stands today. Today, RFCs are managed by the Internet Engineering Task Force.

RFCs are numbered in chronological order, and serve as sort of a timeline of the Internet and its protocols, and their modifications. Many a bar bet has been settle by referring to an RFC index.

RFCs are referred to by their number, and many of these numbers pop up, especially in error messages. For example, mail headers (the information that records how an email was processed) was originally covered by RFC 821, so you’ll often see errors in a mail log that references RFC 821. The same goes for HTML, USENET, DNS, etc… The errors are written that way because the creators want to emphasize that they follow the RFCs, and so should you.

Why are RFCs important? Well, it boils down to communication theory. The Internet at large is basically an anarchy. There are no overriding rules. It’s just data going back and forth. The only way that too entities can communicate with each other is if they agree to. RFCs are a way to manage these agreements. It’s a way to say: “I follow these rules, and if you don’t, don’t expect me to understand what you’re saying.” If you write a program that follows the RFCs properly, you can expect other correctly written programs to understand what’s going on.

Some companies don’t follow RFCs, they try to use their marketing positions and user base to, what one calls “Embrace and Extend” certain protocols. They more or less want to pollute the internet with their own way of doing things, so that they can control who talks to their users, and who their users talk to. Many others are very strict about their interpretation of the RFCs, causing users to get caught in the middle of Open Standards vs Commercial Protocols. This war is hardly limited to RFCs, there are all sorts of standards bodies that companies ignore. Furthermore, RFCs tend to be vague about specific actions. There’s a lot of “you can,” “you should,” and “it is recommended,” talk in most of them. This often leads to arguments about what is allowed and what is not “per an RFC.”

Like I said, anarchy.

Ultimately, RFCs are holy writ to some, and merely “guidelines” to others. Most UN*X Admins follow RFCs and “Best Practices” as best they can. Many others do not. How important is it to follow them? Well, most of the Internet is still run on programs that use open protocols. So far, most initiatives to commandeer them by commercial entities have failed.

I personally believe in open standards, so if you expect to talk to me, or any my systems, you better read the RFC!

I hope that helps.

I recently read an article explaining why FreeBSD was not more popular. The conclusion of said article was that the installer was daunting, and archaic, and that it was too intimidating to utilize. So, basically, whoever wrote this article (I don’t like calling professionals out) didn’t get past installing the operating system. He assumes, that once it’s up and running, it’s the same as Linux. Nothing about the Ports system, nothing about administration. The sum total of his experience was that that installer was intimidating. He went on to state, and I am paraphrasing here, that only old, wizened Unix admins would use FreeBSD, sitting on high from their ivory corner of the office, replete with Star Trek posters, and choice snippets of their homemade 1994 BoFH day-by-day calendars strewn about their desks,  as they are the only ones who would defend such a terrible installer. This is the type that would utilize an operating system that requires disk slices and network configuration. The rest of us “modern” geeks don’t want to bother with such incantations, abjurations and divinations. They just want an operating system that works out of the box. Point-and-click-and-go!

Well, that tells me that you don’t get it. I’m not wasting my time with my installer. You’re wasting your time with yours. And with your point-and-click Linux install, you’ve installed an “operating system” dedicated to wasting time.

It’s all about the futz factor. And you just declared “I live to futz!”

<Here comes the biography>

I am not a wizened UNIX admin. I’m a Macintosh kid. I grew up with GUI objects, and hypercard. I thought that the most efficient way to work with a computer was with a graphic interface. I did some work with DOS, and frankly, thought it archaic, and backwards. Setting base pages for memory, batch scripting, who needed it?

My first experience with UN*X was MKLinux on a Mac LC (the pizza box) I futzed and futzed with it until I got it to boot. No idea what to do with it. 2 years later, my uncle gave me a PII 200Mhz and I put Mandrake Linux on it, to use it as a NAT’ing router and I thought: Cool! Windows sneaked into my life in my late teens, as I could not resist the lure of Counterstrike, Duke Nuke’m and Quake. Still, I enjoyed futzing with Linux. Breaking things, trying to figure out how they were put together, tinker tinker tinker.

I was still mostly a Mac guy when I started my latest job. And, actually, I still am. They were nice enough to furnish a Mac for me, which I happily use, as I like to keep my work environment “tinker free.” My new boss, frowned when he saw that I liked to play with Linux. “Linux is for people who’s time costs nothing.” I didn’t understand.

I fired up an old AMD 2200 based system, and decided to try FreeBSD. My Boss rolled up his sleeves and showed me how to install it. “The Handbook has too many reboots, it wastes time.” he muttered as we plowed through it. At first I was a bit confused, disk slices? Ports? Buildworld? Why??? But the more I worked with it, the more I realized that it was a recipe. A to B to C and you’ve got a fully patched, binary compatible operating system. And it ran Ports!

No hunting down programs that the “distro” didn’t want to install for political reasons; no RPM dependency issues. No graphical nonsense that got in the way. No looking for security vectors that automatically installed. I could have a single task server up in a fraction of the time for a linux install. And if I had to install anything additional, /usr/ports was right there: make config, make, make install distclean. Everything was where it was supposed to be! There was a unified file structure under /usr/local, no /opt /usr/etc /etc nonsense. Clean and neat and ready to rock and roll. Easy to administer, update and upgrade, and NO MAGIC.

What’s more, is that the three versions that have come out since I first started my FreeBSD odyssey have all had more or less the same install template. So I can push out a working server in a fraction of the time I can with Centos or Ubuntu or Gentoo. You want rapid deployment, go with FreeBSD.

Now, Linux is better for some applications. I don’t do Java on FreeBSD unless I know it will work for a certain app. Tomcat, no. I avoid some Perl apps, because BSDPAN is still a little…eh… But all of my LAMP stuff is BAMP, my mail is Postfix/Dovecot and I am happier for it.

I still futz, and when I futz, I tend to play with Linux. I have an Ubuntu workstation at home and an Ubuntu and Centos VM somewhere out there… I’m not opposed to running Linux, but if I want something cookie cutter, reliable and easy to manage, FreeBSD all the way. I don’t care about benchmarks, or raw IO, or sheer device compatibility on my servers. I just want something that doesn’t take up my time. I want more time to futz with other things. Because my time *is* worth something.

Keep your GUI installer. I’ll take my maintainability, upgradability, consistency and, frankly, sanity over it any day.

And before you decry *any* OS/Distro again, Mr Professional Columnist, do more than kick the tires. It’s ok if you don’t like the OS, but when you stop at the installer, and say “this ain’t no good!” you’re just plain ignorant.

Certain…Parties… Have intoned I am  goofy for implementing weird “mail bouncy thing” that is sometimes frustrating and is a silly anti-spam technique. Well, that would be Greylisting, and while it’s weird, it also drops a lot of spam getting through.

Greylisting is a very simple technique. It basically is a daemon attached to database that keeps track of who externally sent mail to who internally. When a new domain/ip-address combination pops up, it bounces that transaction with a temporary, 450 bounce. This is per the RFC, and any properly implemented SMTP server should adhere to it, re-queue the message, and send it again later. If the server sends it before a specified “too early” window (in my case, 2 mins, but that’s fairly aggressive) it’s bounced again. If the message comes back after the “too early” window, but before 24 hours, it’s passed, and an entry is made in the database allowing that address to send mail unhindered for a few days. If enough messages come from the same ip address and the same domain pass greylisting, that whole domain is white-listed.

Greylisting is effective because it keeps non-compliant SMTP servers from sending mail to your server. Most virus infected computers that send or relay spam won’t re-queue messages, or will re-queue them for only the briefest amount of time.

Problems with Greylisting are legitimate, but mis-configured SMTP servers either not re-queuing the messages because they are set to treat 400 series bounces as 500 series, or permanent bounces. Or they re-queue the messages, but report to the original sender that the message bounced.

Yahoo implements a more esoteric set up, where they have 4 servers listed in the MX record, and at any time, any of them will bounce messages. This is another way to test for non RFC compliant servers, as a server is supposed to try all of the MX entries in turn, by weight value. Most virus infected computers won’t do that.

Because some of my users may have problems with receiving mail, I have a web-based interface to the Greylisting daemon’s database that allows me to opt addresses or domains out of Greylisting.

I’ve always run Greylisting, so I don’t have any comparison stats, but this guy does.

Software that I’m using for this:

• SQLGrey
• SQLGrey Web Interface (SGWI)

Very cool for dinking around, PBX in a Flash installs CentOS, Asterisk and FreePBX all in one go, with a couple of extra scripts. It definitely shallows the learning curve of Asterisk. I basically followed this Nerdvittles tutorial, and now I’ve been setting up Queues, IVRs and Voicemail. I still can’t figure out how to get someone to log into a Queue… Oh well. Very fun.

Hey, it works! I’m moving a file (on a GIG-E switch between VLANs) at around 3MBps.

Thanks to http://linux-bsd-sharing.blogspot.com/2008/09/howto-setup-nfs-server-on-freebsd.html

This is a live and learn moment. When I did my last update, I had a lot of old libraries hanging around. I thought that portupgrade would recompile all of my ports, but it didn’t. Most had been recompiled in the interim as I had been upgrading, but Apache, well, Apache upgrades are hardly for the weak. So, when I deleted a bunch of crud laying around. I broke SUEXEC. SUEXEC is what allows scripts to be executed under my various users’ home directories. Well, a recompile and reinstall, and things are working much much faster. Always fix your architecture kids.

I’ve decided to get my learn on, so I’m going through Juniper and Cisco training material. It’s fun, actually. Juniper certs are free, for now, so I’m starting there. Most of their material seems to compare JunOS to IOS anyways so it’s a two for one. Now to get a lab going…