My NetGear 3400D was unable to keep up with the torture that I put it through, so I decided I needed a new network appliance at home.

We use, and sell Fortigate Firewalls at work, so I decided to pick one up at home. This would a) be more robust than a consumer “router” and b) would allow me to work on developing some badly needed network/firewall/vpn knowledge. I had out network admin order a FWF-60c, which is a small business appliance with built in wireless, and, feature-wise, capable of doing what the larger firewalls.

Installing the Fortigate was dead simple, I used the FortiClient software and the wizard to get me off the ground. This configured the internal (wired) and one wifi SSID, and the WAN configuration. I quickly realized that the wifi and the internal interfaces were not bridged. This would present a problem that I would have to solve a little later.

I then decided to test upgrading the firmware from the 4.0 tree to the 5.0 tree. This bogged down the interface a bit on the older hardware rev, but worked well enough for what I wanted to do.

I then refined my policies to match what I had on the NetGear and brought it home.

Performance was astounding.

The first problem I ran into was that I have a first generation Playstation 3, the 20GB edition without wireless. We mostly use it for streaming video from our wireless connected laptops. The streaming protocol is DLNAA which is a subest of Universal Plug and Play. The Fortigate doesn’t support UPnP. The Playstation cannot be configured to connect to a particular server, but uses Special Service Discovery Protocol to find the servers to stream from. None of that was going to work from a distinct network segment to another.

So, out of the box, the streaming wasn’t going to work. I started to look for a solution, but there wasn’t anything online to help me with this, as this was a fairly corner case; people don’t use these for home networking much. I had to do some snooping. Running a few packet captures revealed that SSDP used multicast. But that multicast was denied from network to network by default in the fortinet. Some google-fu taught me how to enable multicast on a fortigate. It has to be done from the CLI, but is like any firewall policy. You’ll need two, one from “internal” to “wifi” and one from “wifi” to “internal”


config firewall multicast-policy
edit 1
set action accept
set srcintf wifi
set dstintf internal
set srcaddr any
set dstaddr any
end
edit 2
set action accept
set srcintf internal
set dstintf wifi
set srcaddr any
set dstaddr any
end


Then, elsewhere in the firewall, put in standard policies to allow the network to pass other traffic to each other, since they are internal, I let the open any ports. I’m sure you can find a list of the myriad streaming ports required if you want something more restrictive. I enabled NAT between the two, I don’t know if that’s necessary, but it worked. And Voila, I can stream! (and do other UPnP things, without the “helpful” daemon.)

Part 2, VPNs

First, my new computer is a MacBook. Mac OS X has a nice little feature that scrubs the files in trash as it deletes them. You can command click on the trash icon and choose “Secure Empty Trash” every time you empty it, or you can go “Finder Preferences”, click “Advanced” and check “Empty trash securely”

Next up, GPGtools. GPG is an open source implementation of PGP security. There are several small encryption related tools, but the killer one is the ability to sign and/or encrypt mail.

Next, I installed TrueCrypt. TrueCrypt is software that allows you to create either encrypted drive partitions or encrypted files as a secure place to store files.

Next, I installed Prey from http://preyproject.com/ This is free software that will help you track down your laptop should someone steal it. You can track three devices for free, including smart phones. I also enabled “Where’s my Mac?” and a cron/reverse-ssh/webserver scheme as other ways to recover my laptop should it grow legs and wander off.

For network access, I use OpenVPN to connect to my firewall, and an IPSEC tunnel to connect to my work firewall. Mike provided me a nifty VPNTracker “Player” that made configuring it a non-issue.

Just a few tips to keep my private life private…

So I did some googling, and found this: http://www.phpclasses.org/package/3613-PHP-Retrieve-and-parse-Apache-server-status.html

Unfortunately, the examples were set to just show the stats of one server, and I had four.

Well, an array, for loop and some php rejiggering, and viola! Active server stats!

Now I can see all the worms attempting to XSS my websites.

Wee!

This is ridiculous. Absolutely unacceptable. Fix it.

Unfortunately, I have shell users who were using mail.chronophage.net as their moniker for pop/imap. This also broke their ability to log in to the antispam area and change their spam settings. I have fixed both of these issues.

Dovecot has a passwd-file facility, which uses passwd formatted files to do authentication. I simply added such a file to my mail cluster, with the shell username and {CRYPT} hashes (out of master.passwd) and added a proxy=y statement and a host=(shell server’s ip)

So now, when they log in, it’ll authenticate them, and then pass the authentication over to the shell server’s POP3 or IMAP server. Since they authenticate locally first, SASL works for sending mail.

This gives them one point for external clients, and still lets them log in via the shell server, or use procmail (via a .forward) if they so choose.

Unfortunately, this means that I have to manually keep password in sync in two places. But my users don’t change their passwords often, and there aren’t too many of them

So, for posterity

mail.chronophage.net (POP3, IMAP, SMTP)
https://mail.chronophage.net (webmail via roundcube)
https://mail.chronophage.net/squirrelmail (webmail via squirrelmail)
https://mail.chronophage.net/antispam (for antispam services)
gaia.chronophage.net (shell server)

*UPDATE* Post approved: http://blogs.iphouse.net/2011/11/10/infrastructure-and-other-games/

Not bad for not having a real load balancer…

This is a four host (with one NFS/MySQL host) against a WordPress Page (this one, in fact) Lots of Caching is enabled, but it’s using fcgid for rendering speed, not capacity. My first host took the brunt of the memory stress for some reason. Perhaps internal connections caused it to be the caching whore.

I suspect some of the trouble is the “stickiness” persistence method of pfSense. If it were pure Round Robin load balancing, I think things would look a lot smoother.

The mail cluster is going well too. Mail is a lot more… finicky, especially when you’re doing spam filtering and greylisting on the same box. I don’t expect that to be as scalable,  but it’s working pretty well. If I had a few more resources, I’d break my servers into at least another tier, my ideal setup is something like this: Frontend->Spam filtering-> SMTP out/Delivery with separate POP/IMAP… But that’s not going to happen with my little (well, medium sized) VDC. Unfortunately, with mail, you have to take a wait and see approach.

So yeah, why clusters?

Well, one, because they can handle more load than a single big server. No matter how beefy you make it, a single LAMP server is going to crap out at around 250 simultaneous connections. Granted, that’s a lot of load, but still. Two, if I take down a server for updates, I don’t take down my website. Three, clusters are a good way to take advantage of a virtualized environment. Each of my servers has the advantage of having a big NAS behind it, and can move from host to host, making my cluster a lot more nimble, and available, than a VPS slice, Container, or Jail residing in a bigger OS.

Clusters are the way to go. Sorry Calypso…

UPDATE:

Switched the cluster to apache2-mpm-worker. Here are the results!

I’ve been waiting for my work to release a its new product. I’ve been waiting, politely, for my boss to blog about it. I’ve been waiting to show off this new product.

I’ve been working on provisioning, and working with customers on beta testing the new product. I’ve been working on templates, and auto install media, to make everyone’s life easier. I’ve been working on documentation for customers.

I’ve been waiting for, and working on, a VMware vCloud Director based product known as vmForge VDC.

This is cool stuff!

It combines the power of co-location with the flexibility of virtualization and the ease of cloud based resource management. It’s reliable, consistent, and powerful.

It’s also straight forward. Virtual machines are organized into “vApps” which are logical containers that can be used to start, stop, isolate and template groups of VMs.

To test things out, I’ve decided to move Chronophage’s services into a virtual data center. Unfortunately, I can’t use up one of our virtual domains on our physical firewalls (that is a little too expensive of a comp for an employee) so I’m routing down a /28 to a /30 on a pfSense virtual machine that’s acting as my firewall, router and loadbalancer. I set it up, templated it, and filed it into my VDC’s catalog just in case I need to re-deploy it. I’m running pfSense 2.1-Development because I want access to ipv6.

The loadbalancing is fairly simple in pfSense, round robin style (or fail over) with monitoring. But it works natively for ipv6 (in 2.1) and it has some limited persistence, which it calls “sticky”. If there are any open states between an outside ip A, and an internal ip B, it’ll send further traffic for loadbalanced services from A to B

The first thing I decided to deploy was a webcluster. Right now, this consists of two front end machines running php via fcgid and a backend server running nfs and mysql. Using fcgi + suexec (custom), every site has its own user, so I set up NIS on the internal network to keep everything in sync. I set up vsftpd on the nfs/db server, so that the files can be updated by each user.

After verifying everything was working (with some telnet love) I set up a virtual ip for loadbalancing, and cut over DNS. Everything was flying! I then added a ipv6 block, set up a virtual ip for ipv6 pointing to the servers ipv6 addresses, and now I have native ipv6 from end to end!

Next up will be mail, which is a little trickier to “clusterize” because there is much more writing involved. I plan to have two front end machines and a backend machine, plus a shell machine for people who want to have a unix environment, file storage and local mail.

Now, not to bring pricing into this, but this setup would cost me around $450/month, including bandwidth, if I weren’t getting it compensated. That’s right around the price for the *space* needed to run all of those servers (barely) in most data centers, not counting the bandwidth, power, and the cost of purchasing the physical servers.

The best thing about this, I built it all from my desk, in a few hours. And most of that was trial and error while working on a few things that I wasn’t very familiar with. Now, you’re reading this on my web cluster!

Co-location is dead, the cloud is confusing, but this, this is dead simple!

I like dead simple.

Someday I’ll tell you about implementing other cloud solutions. Not so simple