Chronophage.net Blog

So I put a hypervisor in your hypervisor, so you can virtualize while you virtualize.

Ubuntu Running VirtualBox, running Windows 7, running vSphere Client connected to vSphere hosted on ESXi running two ESX hosts and Nexenta, together running an Ubuntu Netbook VM on shared storage.

This shows the storage, both hosted in the lab enviroment (nexenta, sexy) and external (datastore(1) CALLY – an NFS server)

Elastic Sky X. That’s what ESX stands for. Crazy, right?

Well, more playing means more caveats

First, something I forgot to mention yesterday. To get ESX working in Ubuntu, Workstation needs to be able to put the vmnet interfaces into promiscuous mode. That requires allowing the user or group that you use to start Workstation to have read/write permissions over the vmnet devices in /dev. A simple chmod will do the trick.

Now, back to our story…

I decided to try to add some additional network paths to hopefully improve I/O. In the last post, I mentioned the VM within a VM was *painfully* slow. Well, change *painfully* to *detrimentally*. The whole environment was prone to timeouts, something that vSphere doesn’t always seem recover from gracefully.

This is understandable. Enterprise software expects an enterprise environment. Something that day hackers should appreciate. (see the perpetual “Linux Vs BSD debate” ) Dinking around is going to generate problems that one would not see in a production environment. It’s a pet peeve of mine when people review software, give it inadequate resources, and complain about the end results.

That being said, I had problems.

Adding a second nic to the ESX VMs required 2 restarts per host. Arbitrarily. The first host’s second NIC did not work via DHCP (again, not something I would rely on in production) and had to be manually configured. The second host dropped all of its routing information, and again, needed to be set up manually. I attempted a restart on both, just to see what would happen. The first host again had to be set up manually. The second came back fine or so I thought…

After playing the restart shuffle, I had both NICs in place. I decided to try creating a distributed switch. Creating a distributed switch is a three part process, that requires two views. First you create the switch in the Networking view, assigning an uplink NIC to it. Then you configure the ports. Then, back in “Hosts and Clusters” you have to add the port type (VMkernel, Service Console or VM Network) After setting it up, the distributed switch propagates to all of the hosts that you’ve assigned to it. It’s more or less a way to have a consistently configured network across multiple hosts. Very slick, but very expensive. I have a feeling that this methodology will be come more standard amongst VMware’s competitors, and will be rolled into the product at large, or at least lesser license levels. Still, networking, as my boss is prone to say, is like plumbing. If it is done correctly, there’s no need to redo it.

Unfortunately, my second ESX host failed to take the Distributed Switch configuration due to it timing out. This caused the host’s network information to become broken. How broken? Well, I poked around for a bit, and decided it was faster, and easier to install a new ESX VM. If I have time, I’ll look at the host and figure out what’s wrong, as learning to troubleshoot vSphere is a secondary goal of mine for this project. However, for now, this new ESX VM gave me an opportunity to play with the second big “Enterprise Plus” feature. Host Profiles.

Host Profiles allow you to create templates based off of existing hosts. You can then store the Profile, update it, and apply it to other hosts. After bringing the new ESX server online, I created a profile from my original ESX host. I then put the new one into Maintenance mode, attached the profile, and applied it.

After a quick summary of what changes it was going to make, It started the application process. I don’t know how it ended, as it was midnight this morning, and It was well past time to hit the hay. I assume things went well

It was interesting to see in a few lines, all that I’ve done to configure my lab machines, and it will be nice not having to do it again. Although configuring ESX is hardly an arduous chore, (well, it’s slightly painful with my lab’s limited resources) I can see how configuring more than a handful at a time would be obnoxious. Host Profiles aren’t exactly necessary as, like networking, you’re not going to constantly provision new servers. If you’re in an environment that does, you’re probably a large company that can afford the licensing fees, and the cost/benefit factor is in your favor. Still, it would be nice if you could simply copy the config from one server to another in lesser editions of vSphere, without storing the profiles or being able to update them. That would be a good compromise, and I think will become standard with VM infrastructures across the board. Red Hat does something similar with it’s OS centric Satellite/Spacewalk servers, which handle both initial configuration and update management. It’s not nearly as slick, as you do need to “script” installations.

Well, that’s day two of the “fully functioning” lab. We’ll see if the Host Profile worked on the new ESX VM

Requirements:
VMware vSphere, lots of RAM, a decent amount of disk space, a fairly recent copy of 64bit Windows (I used Server 2008 R2) ESX and vSphere Server iso and exe files. Iron will. Patience. Some sort of NAS distribution (I used FreeNAS.)

So I wanted to set up vSphere. We use it quite a bit at work, but I wanted to try it in a full, yet simulated environment. I have a desktop that I bought in ’07 for gaming. Although it’s a bit long in the tooth, it had enough CPU (e6600) to get the job done. Also, it was before Intel released core2duo chips that lacked VT extensions. I added 4GB of RAM for a total of 6GB. I had installed Ubuntu on it, as I lacked an extra Windows license, and frankly, it’s good enough.

I installing a demo copy of VMware Workstation, as I like Fusion at work. Workstation is a seamless interface that allows juggling multiple VMs with an intuitive network set up and centralized management. I may buy it.

I first set up two ESX hosts. A very painless and quick install. Then I installed Windows 2008R2. Installing vSphere on it took a bit of time, but was relatively straight forward. I didn’t feel like futzing with SQL server, so I installed SQL Express via the default setup. Again, very straightforward. If this were more of an active lab, or production environment, I would’ve installed SQL server, as SQL Express can’t hold more than a few hosts’ worth of data. vSphere generates a TON of statistical and historical information.

After adding the hosts, and putzing around a bit, I installed VMware Update Manager, and VMware Converter. I’ll play with VUM, but probably not Converter.

To get vMotion to work, I had to install a VMKernel port. iSCSI and NAS required a Service console port. vMotion had to be enabled on the port, and an iSCSI software hba need to be enabled.

After getting the front end systems up, I installed FreeNAS in another VM in Workstation. I assigned it in 2 80GB, thinly provisioned disks for NAS exports, and 10 8GB disks for various other applications. In FreeNAS, one can add disks on the fly. After figuring out that FreeNAS did *everything* in a hierarchical manner, setting up NFS exports was trivial.

Back in vSphere, exposing the ESX hosts to the NFS mounts was trivial, and I utilized all the space as two 80GB datastores. The I/O hit wasn’t trivial. The lack of network segmentation, even virtually, takes its toll.

iSCSI was a bit more difficult. FreeNAS had a few more levels of configuration than I expected, so I missed some steps. Basically, you set up an Extant, and then set up a target to it, and then hit apply. The last step bit me a few times

I exposed 2 disks as unformatted block devices, and created two files on the UFS formatted file system to the other two. Setting up a software RAID is pretty easy in FreeNAS, but I was just poking around, so I didn’t care about redundancy. Either option seemed the same. Curiously, each extent showed as a separate “disk” each with LUN 0. I’m sure there’s a way to change it to one “disk” with several LUNs, but I’m playing with vSphere, not FreeNAS.

Back in vSphere, I had to enable iSCSI, point the hosts’ iSCSI HBAs to the NAS/SAN VM. I had four LUNs available: two using the “write to file” method, and two as block devices. I set up the first Datastore with the “file” LUN first, and extended it with the last block device LUN. I did the inverse with the remaining two. That gave me two 16GB Datastores. Again, very easy.

FreeNAS gave me trouble with iSCSI and ZFS. I was able to set up a ZFS/RAID 5 pool, but the native method to attach it to iSCSI didn’t seem to work. I’m sure I’m just missing something, the the volume wasn’t available to attach to an iSCSI extent. Maybe I’ll figure it out, maybe I won’t. Again, I’m more interested in vSphere, and not FreeNAS.

Seeing as I was running my virtualization environment in an virtualized environment, I didn’t have VT extensions available to my ESX hosts. That means that they could only run 32bit VMs on them. Fine for testing, but I only had 64bit ISOs. So a quick download of Ubuntu Desktop Edition, 32Bit, and I had an Ubuntu VM running in an ESX VM running on Ubuntu A surreal experience. I don’t think VUM updates Ubuntu, so I’ll have to dig around for an old XP iso. I’m looking forward to playing with VUM, and Distributed Switches.

Things I would do differently:

OpenFiler, which I tried first, did not work intuitively. None of the HOW TOs I found online matched the interface I was looking at, and the instructions cost 40 quid (is that how you say that, I am very not British) . No thanks.

If I had more resources, and time, I would set up things more like what I would expect to see in an enterprise environment. More switches, better segmentation, more redundancy in the storage, etc. My lab would get an F in Infrastructure Design. As it is though, the VM within a VM is *painfully* slow.

vSphere is fun, and I’d love to learn more. Soon, due to Workstation’s demo license expiring, I will have to dismantle my lab. Perhaps I can recreate it with KVM and virt-manager (ouch!)

Updates may follow.

“Sudo’s command matching routine expects actual commands to include one or more slash (‘/’) characters. The flaw is that sudo’s path resolution code did not add a “./” prefix to commands found in the current working directory. This creates an ambiguity between a “sudoedit” command found in the cwd and the “sudoedit” pseudo-command in the sudoers file. As a result, a user may be able to run an arbitrary command named “sudoedit” in the current working directory. For the attack to be successful, the PATH environment variable must include “.” and may not include any other directory that contains a “sudoedit” command.”
(From http://portaudit.FreeBSD.org/1a9f678d-48ca-11df-85f8-000c29a67389.html)

I actually read about this on Full Disclosure. This is very similar to an earlier exploit. Sudo is a great tool, but you always have to be *very* careful who you give sudo access to.

So, some of my users are avid World of Warcraft players. They’ve been complaining that they keep getting phishing schemes in their email accounts. Since Bayesian Filtering isn’t catching on, I think it’s time for custom SpamAssassin rules.

First, a quick scan of SARE rulesets to see if one fits the bill. I picked the the “Forged” rules simply because they are logical and simple. I like logical and simple

Then some quick adapting…

and this is what I came up with:

header __RCVD_WOW Received =~ /worldofwarcraft\.com/i
header __FROM_WOW From =~ /worldofwarcraft\.com/i
uri __URI_WOW /worldofwarcraft\.com/i
meta CUSTOM_FORGED_WOW (__FROM_WOW && __URI_WOW && !__RCVD_WOW)
score CUSTOM_FORGED_WOW 1.4

I have similar rules for blizzard.com and battle.net. What I’m looking for is the domain in the “From” field in the headers, and the domain in the body of the message but the domain NOT in the “Received” field. Most phishing schemes have a hotmail/yahoo/gmail address in the “Received” (and the “Reply To” field)

The only caveat I see with this, is that the base64 encoding of the message may screw up the body check, but I’m almost positive Spam Assassin decodes the message first.

I assigned a relatively small score of 1.4 to the messages, just to see if they show up in the “X-SpamScore” header. Remember, only one should hit at a time.

We’ll see if that works.

An unfortunately common trend across the IT world is this: Your IT infrastructure is running great and everyone is happy. The budget cuts have to happen and the boss looks around, and decides that their good, but expensive IT professional is no longer worth paying. So they replace him or her with someone cheaper, with less experience, and problems set in. Often major problems. Usually, at this point, the company goes back to their original IT professional with their hat in hand, or their IT offices start to resemble a by-the-hour hotel.

It’s easy to want to compare IT to Sales and Marketing. But those traditional roles have a direct correlation to performance and results. Sales does well, money comes through the door. Marketing does well, and your company is on the lips of Jane Q Public. If IT performs excellently, nothing happens.

Nothing.

It’s hard to justify nothing. Nothing doesn’t make the books.

So, the person in charge of the purse strings looks around, sees someone who apparently does nothing, and gets rid of him or her.

Then something happens. The network goes down. The un-patched server is compromised. Money starts flying out the door. Consultants are called, the emergency fixes is done. The new IT guy or gal is fired, and the boss goes looking for a better qualified IT professional. Lesson learned, right?

Unfortunately, not. Often, the new IT Professional comes on, gets everything working again, and what does the boss see? Nothing.

Wash, Rinse, Repeat.

There are really two ways a professional deals with this: 1) He or she rides out the cycle with his or her reputation intact, but has to deal with scrambling for a new job more often than not, or 2) He or she implements Operation Job Security.

Operation Job Security is a simple formula really. Implement arcane solutions that are fragile, and prone to break often, but not too often. Don’t document anything. And play the martyr whenever you have to fix anything.

The last time I talked to a small business owner who loved his IT guy, I was at a bar after work. The conversation went something like this:

“I love my IT guy,” he tells me, out of the blue. Well, not really. Our conversation went from the weather, to our jobs. The standard Midwestern small-talk. He owns a small manufacturing business, I do IT.

“Anytime anything breaks, my guy is there, 2 am, whenever. He’s amazing.”

“Oh?” I take a sip of my Guinness.

“How often do things break?” My question hangs in the air while he thinks about it.

“About once a quarter.” he says.

It doesn’t click. Yet.

“Reliably?” I ask him, with a touch of irony.

“Well, yeah…” There’s a moment of awkward silence. Realization sets in.

“Oh.” he says.

“Oh.” I say.

Oh is right. Mr “I love my IT Guy” is a victim of a master of Operation Job Security.

The word “victim” is right. I don’t blame IT Guys or Gals who implement Operation Job Security, we all have to eat and pay bills, but it *is* a con game.

Sabotage is not cool, it’s not professional, it’s not ethical, and ultimately, it gets you nowhere. It’s predatory, and violates the trust between a client and his or her paid professional. It breeds resentment, on both sides of the equation, and makes the whole industry look bad.

Take the high road. Take pride in your Nothings. Do it right.

Then again, that’s easy enough for me to say.

[Originally Appeared 02/04/2010 blogs.iphouse.net]

One of the many terms you’ll hear thrown around an internet service provider is Request For Comments, aka, RFC: “This isn’t per the RFC!” or “We follow the RFC!” or “Read the RFC!” So what is an RFC, and why do you want to know what it says.

RFCs are, in a nutshell, the description of how a program, or procedure should work. The history of RFC is long and boring, but basically, they’ve been around since the ARPANET Project began, as written or typed memo that were literally Requests for Comments, open ended questions that someone wanted to solicit answers to. As ARPANET grew, RFCs became the standard way to record procedure, and a way for people to implement the fundamental technologies that make up the Internet as it stands today. Today, RFCs are managed by the Internet Engineering Task Force.

RFCs are numbered in chronological order, and serve as sort of a timeline of the Internet and its protocols, and their modifications. Many a bar bet has been settle by referring to an RFC index.

RFCs are referred to by their number, and many of these numbers pop up, especially in error messages. For example, mail headers (the information that records how an email was processed) was originally covered by RFC 821, so you’ll often see errors in a mail log that references RFC 821. The same goes for HTML, USENET, DNS, etc… The errors are written that way because the creators want to emphasize that they follow the RFCs, and so should you.

Why are RFCs important? Well, it boils down to communication theory. The Internet at large is basically an anarchy. There are no overriding rules. It’s just data going back and forth. The only way that too entities can communicate with each other is if they agree to. RFCs are a way to manage these agreements. It’s a way to say: “I follow these rules, and if you don’t, don’t expect me to understand what you’re saying.” If you write a program that follows the RFCs properly, you can expect other correctly written programs to understand what’s going on.

Some companies don’t follow RFCs, they try to use their marketing positions and user base to, what one calls “Embrace and Extend” certain protocols. They more or less want to pollute the internet with their own way of doing things, so that they can control who talks to their users, and who their users talk to. Many others are very strict about their interpretation of the RFCs, causing users to get caught in the middle of Open Standards vs Commercial Protocols. This war is hardly limited to RFCs, there are all sorts of standards bodies that companies ignore. Furthermore, RFCs tend to be vague about specific actions. There’s a lot of “you can,” “you should,” and “it is recommended,” talk in most of them. This often leads to arguments about what is allowed and what is not “per an RFC.”

Like I said, anarchy.

Ultimately, RFCs are holy writ to some, and merely “guidelines” to others. Most UN*X Admins follow RFCs and “Best Practices” as best they can. Many others do not. How important is it to follow them? Well, most of the Internet is still run on programs that use open protocols. So far, most initiatives to commandeer them by commercial entities have failed.

I personally believe in open standards, so if you expect to talk to me, or any my systems, you better read the RFC!

I hope that helps.

I often get questions about SEO and what can be done to enhance a web page’s rankings. SEO is definitely something to be aware of, and there’s quite a bit “known” about the way certain web browsers rank their pages, but it’s hardly a system that you can easily or reasonably game. (Yes yes, there are some obvious holes, but don’t expect them to last.)

SEO is a knowledge based cottage industry, as such, it attracts two types of people. People who know what their doing, and bullsh*tters. It’s easy enough to spot the latter, people who “specialize” in SEO will at best inflate their importance, and at worse, sell you a bunch of hot air. People who give SEO advice in addition to other web design tips, tend to be legitimate.

Search Engine Optimization is an inexact science. Search Engine algorithms are not publicly known, and are prone to changing often. Now, information leaks, and people can compare notes, so a lot is known about how search engines seem to work, kinda. However, most of these SEO “tricks” are really common sense. Most things boil down to things like: Make sure you don’t have a ton of cross links, don’t have a bunch of throw away domains, have an older domain, get your stuff linked to from reputable communities and pages, etc.

All of this falls under the purview of web design.

A good web designer knows the basic SEO ins and outs, and can, you know, design a decent looking website. A good web designer will build a functional, well laid out, logical website that people will want to come to, and people will link to and recommend. A good web designer designs good websites. Designing good websites means that you keep an eye on the SEO stuff, because SEO more and more means “Best Practices.” Bad web designers make bad websites. Bad websites get bad page rankings.

So then next time someone claims to be a SEO “Expert” who will O you so hard with the SEs you’ll be crying, and will guarantee a top ten spot, kick’em to the curb. Because he or she is selling virtual snake oil.

I recently read an article explaining why FreeBSD was not more popular. The conclusion of said article was that the installer was daunting, and archaic, and that it was too intimidating to utilize. So, basically, whoever wrote this article (I don’t like calling professionals out) didn’t get past installing the operating system. He assumes, that once it’s up and running, it’s the same as Linux. Nothing about the Ports system, nothing about administration. The sum total of his experience was that that installer was intimidating. He went on to state, and I am paraphrasing here, that only old, wizened Unix admins would use FreeBSD, sitting on high from their ivory corner of the office, replete with Star Trek posters, and choice snippets of their homemade 1994 BoFH day-by-day calendars strewn about their desks,  as they are the only ones who would defend such a terrible installer. This is the type that would utilize an operating system that requires disk slices and network configuration. The rest of us “modern” geeks don’t want to bother with such incantations, abjurations and divinations. They just want an operating system that works out of the box. Point-and-click-and-go!

Well, that tells me that you don’t get it. I’m not wasting my time with my installer. You’re wasting your time with yours. And with your point-and-click Linux install, you’ve installed an “operating system” dedicated to wasting time.

It’s all about the futz factor. And you just declared “I live to futz!”

<Here comes the biography>

I am not a wizened UNIX admin. I’m a Macintosh kid. I grew up with GUI objects, and hypercard. I thought that the most efficient way to work with a computer was with a graphic interface. I did some work with DOS, and frankly, thought it archaic, and backwards. Setting base pages for memory, batch scripting, who needed it?

My first experience with UN*X was MKLinux on a Mac LC (the pizza box) I futzed and futzed with it until I got it to boot. No idea what to do with it. 2 years later, my uncle gave me a PII 200Mhz and I put Mandrake Linux on it, to use it as a NAT’ing router and I thought: Cool! Windows sneaked into my life in my late teens, as I could not resist the lure of Counterstrike, Duke Nuke’m and Quake. Still, I enjoyed futzing with Linux. Breaking things, trying to figure out how they were put together, tinker tinker tinker.

I was still mostly a Mac guy when I started my latest job. And, actually, I still am. They were nice enough to furnish a Mac for me, which I happily use, as I like to keep my work environment “tinker free.” My new boss, frowned when he saw that I liked to play with Linux. “Linux is for people who’s time costs nothing.” I didn’t understand.

I fired up an old AMD 2200 based system, and decided to try FreeBSD. My Boss rolled up his sleeves and showed me how to install it. “The Handbook has too many reboots, it wastes time.” he muttered as we plowed through it. At first I was a bit confused, disk slices? Ports? Buildworld? Why??? But the more I worked with it, the more I realized that it was a recipe. A to B to C and you’ve got a fully patched, binary compatible operating system. And it ran Ports!

No hunting down programs that the “distro” didn’t want to install for political reasons; no RPM dependency issues. No graphical nonsense that got in the way. No looking for security vectors that automatically installed. I could have a single task server up in a fraction of the time for a linux install. And if I had to install anything additional, /usr/ports was right there: make config, make, make install distclean. Everything was where it was supposed to be! There was a unified file structure under /usr/local, no /opt /usr/etc /etc nonsense. Clean and neat and ready to rock and roll. Easy to administer, update and upgrade, and NO MAGIC.

What’s more, is that the three versions that have come out since I first started my FreeBSD odyssey have all had more or less the same install template. So I can push out a working server in a fraction of the time I can with Centos or Ubuntu or Gentoo. You want rapid deployment, go with FreeBSD.

Now, Linux is better for some applications. I don’t do Java on FreeBSD unless I know it will work for a certain app. Tomcat, no. I avoid some Perl apps, because BSDPAN is still a little…eh… But all of my LAMP stuff is BAMP, my mail is Postfix/Dovecot and I am happier for it.

I still futz, and when I futz, I tend to play with Linux. I have an Ubuntu workstation at home and an Ubuntu and Centos VM somewhere out there… I’m not opposed to running Linux, but if I want something cookie cutter, reliable and easy to manage, FreeBSD all the way. I don’t care about benchmarks, or raw IO, or sheer device compatibility on my servers. I just want something that doesn’t take up my time. I want more time to futz with other things. Because my time *is* worth something.

Keep your GUI installer. I’ll take my maintainability, upgradability, consistency and, frankly, sanity over it any day.

And before you decry *any* OS/Distro again, Mr Professional Columnist, do more than kick the tires. It’s ok if you don’t like the OS, but when you stop at the installer, and say “this ain’t no good!” you’re just plain ignorant.

Very cool for dinking around, PBX in a Flash installs CentOS, Asterisk and FreePBX all in one go, with a couple of extra scripts. It definitely shallows the learning curve of Asterisk. I basically followed this Nerdvittles tutorial, and now I’ve been setting up Queues, IVRs and Voicemail. I still can’t figure out how to get someone to log into a Queue… Oh well. Very fun.