So anyways, enough waffle here. Those of you who are interested, should follow this link: my step-by-step guide on how to set up a FreeBSD PV guest from scratch, including pygrub support. The tutorial will walk you through all the steps required from setting up an empty stub, over creating a HVM guest, to transforming that into a PV guest (or even hybrid, if you wish).

What is GlusterFS, and what is it not?

You can think of GlusterFS as a RAID device, which works across the boundaries of a single physical disk array. Take RAID-1 for example, which mirrors data between two identical disks. In GlusterFS’s jargon, you run two bricks in replicate mode, where a brick is defined as storage in general terms; it can be an array of disks (which could use RAID), a single disk, a partition, a directory. Anything that can be mounted into your filesystem hierarchy qualifies as a brick. The key feature of GlusterFS is now, to treat bricks on different physical machines as one volume, which can be accessed by any number of clients. It can be mounted either via Fuse/GlusterFS client, or even via NFS or CIFS/Samba.  You can use RAID-0 style striping for read speed, RAID-1 style mirroring for real-time replication, RAID-10 for both, or you can go beyond any of those and spread the stripes or mirrors across any number of bricks. 4-node replication? No problem at all. GlusterFS gives you truly enormous flexibility and performance when it comes to making large amounts of data available across multiple nodes.
Since version 3.2 (if I’m not mistaken), they have even added GeoReplication, which allows a Master/Slave setup, where the slave can be a local or remote site. Be it for backups or to have a standby version of your application in a different geographical location… it’s possible. Due to the fact that GeoReplication does not require locking or synchronous replication, the network speed to your remote site isn’t that important either. It copes well with it.

This sounds very different than for example a DRBD/GFS2 or DRBD/OCFS2 setup, doesn’t it? And indeed it is! GlusterFS, unlike DRBD, is not providing a block device. What it means is that it compares hashes of files, and if files on nodes differ (for example after a failure), it will copy entire files across, not only the changed blocks. In normal day-to-day operation that’s not a big problem, in particular as you get a lot of flexibility, which is unmatched by other solutions. Where it does make a difference is during recovery. More on that in the Caveats section.

A variety of different connectors

I mentioned earlier that you can use a couple of different ways to connect to your GlusterFS volumes. First, there’s their own GlusterFS client, which uses the kernel’s Fuse layer. This client is Gluster’s recommendation, if your workload requires a high amount of fast write operations. If your workload is more about reading small files quickly, they recommend NFS. (The NFS server is part of the glusterd daemon, which serves the volumes to the clients.) Samba/CIFS is probably mainly targeting Windows clients.

All these connectors have their advantages and disadvantages. You want to test that thoroughly for your particular workload. Also, in SELinux environments, you will require some tweaking of your policies, if you use the GlusterFS client, whereas NFS is a lot more straight forward (don’t forget that apache needs to be allowed to access NFS directly if that’s your intention; setsebool -P httpd_use_nfs=on is your friend). I know most people find it easier to switch off SELinux altogether, but for me personally that is never an option. I’d rather spend hours tweaking the SELinux policies, if necessary. Anyhow, the bottom line is that both NFS and CIFS make GlusterFS very attractive for platforms beyond Linux. FreeBSD for example, although I’m not sure if the native client has reached a production-ready state there yet; I shall give that a spin soon, and in the meantime NFS will do.

Performance

As a rule of thumb you can say that high-availabily, robustness, scalability etc always come with a downside: write performance. During write operations, all nodes need to be kept in sync, which means that the weakest “link” (or slowest disk for that matter) together with some locking and network/protocol overhead determines the actual write speed. That is normal. (Note: pure throughput must not be confused with the time it takes to actually be able to access a file on a different node than it was written to)

For that reason you can never expect a high availability file system to solve all your problems. There’s no such thing as “one size fits it all”. Your application need to be cluster/HA aware. In practice that means you will have to select carefully which type of information you store where. This is of course true for GlusterFS, too. However, when it comes to read performance, GlusterFS is actually very fast. Not as fast as a local block device, obviously, but personally I wasn’t able to tell the difference between native NFS and Gluster’s NFS implementation. The GlusterFS client (fuse/glusterfs, not NFS) however seems to be a little bit slower reading data, while being faster writing. It really depends on your workload. Bottom line is: GlusterFS is fast and flexible, which alone is a big plus over many other solutions. For maximum read performance you can of course use stripes (data scattered across multiple nodes), which the glusterfs client connects to simultaneously. It’s kind of obvious that in particular big files benefit from such a setup.

Caveats

If you intend to deploy GlusterFS, you better plan a serious amount of time for the first tests, integration into your setup, including benchmarks and failover. GlusterFS is powerful and not too difficult to get started with, but you’ll soon run into various rather specific questions, which aren’t documented well (or not at all). Quite frankly the online documentation is poor, or rudimentary. Obviously Gluster, a business, wants to sell their expertise, and there’s nothing wrong with it. So be prepared to browse mailing list archives or hang out in #gluster in irc.freenode.net or so.

GlusterFS has matured a lot over the last years, and you certainly don’t need to be worried about losing data (after all it’s filesystem based and you can copy anything out of the bricks’ directories directly, if you wish). However, some major issues and pitfalls still exist.

• If you reintroduce or replace a node, which was either faulty or offline for a while, the self-healing will transfer entire files back from up-2-date nodes onto the reintroduced one. This consumes a lot of network bandwidth, and even worse, CPU load (possibly due to the hash comparison). If a GlusterFS brick lives on a box together with other services, you will experience a significant performance hit.
• Large files are locked while being replicated. In practice that means that you really can’t use GlusterFS as a backend for VMs at the moment, unless recovery always happens in a controlled manner at times where you can afford to shut down running VMs for the entire duration of the healing. That somehow defeats the purpose of a high-availability storage cluster.
  However, a GlusterFS engineer has told me earlier today on irc.freenode.net that this issue will be tackled in GlusterFS 3.3, if not earlier. Only a question of months, I suppose.
• You absolutely must synchronise the system time of all bricks. If you’re not doing that already anyway, do it before deploying GlusterFS. (use NTP for your own sanity)
• Make sure that the bricks of one volume are of identical size and that you don’t by mistake fill the disk space by other means. I had a situation the other day where I wanted to replace a brick; what I didn’t realise first was that someone set a disk quota on the new brick. Consequently it stopped writing long before all data could be copied. However, GlusterFS did not warn me, nor did it report an error; it actually confirmed successful migration, although only 1/3 of the files were transferred!
  Clearly the lack of accessible disk space wasn’t GlusterFS’s fault, and is probably not a common scenario either, but it should spit out at least an error message. Imagine what would have happened if I had taken the other node offline after allegedly successful migration! Total mess.

Presumably none of these things would have happened, if I had taken their commercial offerings.  Those of you who prefer D.I.Y., better be prepared to spend a serious amount of time to fit it into your use-case and more importantly… monitor it closely!

Summary

GlusterFS has made a lot of positive progress over the last 2-3 years. It’s very easy to get started, especially on RHEL/CentOS, and it offers enormous flexibility and opportunities. The new CLI makes basic configuration much much easier than it used to be before. With a few simple commands you can create your volumes (on multiple servers, aka “peers”, simultaneously). You could say that it’s actually fun to use GlusterFS!

However, if you (like me) are looking at GlusterFS as a backend for Xen or VMware VMs in order to facilitate live-migration and resilience, you will probably need to wait for version 3.3, unless controlled recovery with planned downtime is an option for you. Might be worth keeping an eye on their Git repository (I certainly will). While using it to serve files for all sorts of things already, I’m really looking forward to using it as a backend for Xen soon!

Version 3.3 brings some other new promising features, too… Unified storage, object storage… I see memcached on the list of dependencies… looks promising. Beta 1 is out, by the way.

So where’s the but? Here it comes: …but if something unexpected happens, you are seriously screwed. Here are a few examples from the past couple of months.

Changing a NIC, which is also management interface, of a pool server — This was about the worst nightmare I’ve ever had. What you’d expect to do is: shutdown the machine, open it, replace the NIC, close it, switch it on again, wait for it to boot and start the VMs, done. What really happened is: I had to actually wipe and re-install the whole box, because there was apparently no documented, reverse-engineerable, or otherwise known way to just simply change the MAC address somewhere, because that is managed by the pool master. Now, as the NIC was broken, the master wasn’t able to communicate with the pool server any more (not even on the second NIC, because that was not the management interface). Attempts to change it failed. Not even the “xe” tool was functional any more, so I couldn’t really gather the UUIDs in order to search through configurations etc. The master refused to talk to the pool server, and the pool server with the broken (and afterwards replaced NIC) refused to let me change anything, because that should be done on the master. Catch 22.

I consulted the official support forum, but nobody knew an answer there either. I’m sure there is a way to change it easily. After all it’s a Linux box with a modified Xen, but still not an unaccessible blackbox. Hang on… actually it felt a bit like that. I would like to think that Citrix certainly knows an easy solution, but as I’m not paying thousands of Pounds for a product, which is almost entirely based on free software, they of course kept quiet. (The bloody toolstack, which complicated things, is their own development, by the way.)

End of that experience was that I had to remove the server from the pool (XenServer would then wipe the box, so you can’t re-join the pool later, either… awesome). After a clean setup and restoring all the VMs from previously created snapshots, the machine was finally able to join the pool. That was 6 hours after the NIC broke. Fortunately all VMs have an identical twin running on another machine, so it didn’t cause downtime (except a few minor hiccups while I was fiddling about with network settings). Otherwise all websites/applications would have been offline for 6 hours.

Without the XenServer toolstack, I could have resolved the issue within 10 minutes, which includes all of the steps mentioned earlier (what I would have expected).

I learned my lesson from it. As live-migration of VMs isn’t really necessary in most cases (my customers’ applications don’t benefit from it), it’s actually better to not form pools of your servers. Disconnected standalone servers are a lot easier to maintain and you don’t risk side-effects with pool members, because there aren’t any. The only real downside is that VLANs need to be configured individually on each server. Same applies to shared resources (NAS etc). But that’s fine.

Another almost unbelievable example is deleting snapshots. I create them all the time, because if something goes wrong, or someone breaks a VM setup, you want to be able to roll back to a previous version. Snapshots are one of the biggest advantages of virtualisation. A whole VM can be brought back to an older state within seconds. Or you can export it and reimport it elsewhere, clone another instance from it, work there, swing later. Anyways, if you use that feature often, it fills your disk (even the huge disks you get nowadays). So you regularly delete them and get your space back. Right? Nope, wrong. With XenServer you may or may not get your space back. When your monitoring tells you that you are running out of disk space, although you haven’t done anything but rotating snapshots in a while, you scratch your head in disbelief. Well, at least I did. Unfortunately, the official documentation confirms my observations. When I first read that reclaiming space causes downtime, I wasn’t sure if laughing or crying was the best course of action.

In a production environment, you can’t just go ahead and suspend VMs just to get space back. Even if you only reduce performance (without causing downtimes, as we’re running twins of everything), you need to make affected customers aware of it. And how do you explain that? “Sorry, Sir, I need to suspend your service, because I need to delete old snapshots.” They’ll think you’re taking the piss.

Again, this “feature” is brought to you by Citrix’s toolstack, not Xen. If I decide to delete an LVM-based snapshot of a running VM on Xen, I can do that any time. No need to suspend anything or to manually reclaim free space afterwards.

My favourite subject is VLANs. I don’t know how many hours I’ve wasted trying to find what I did wrong, just to figure out in the end that it was not my fault… Citrix apparently manipulated the bridge code and never really tested it. You have to actually install ebtables (iptables for bridges, if you will) to work around that issue. I observed exactly the same thing as the poster there, and many others did, too. Their forums are full of problems related to VLANs and NIC bonding. Problems get worse with two NICs. VLANs may work out-of-the-box on both, only one, or none of the NICs. Apparently it depends on the used NIC (well, I’m assuming here that nobody uses old NICs without VLAN support any more nowadays), which of the NIC is management interface, and a couple of other factors like weather, mood etc.

Once you know about the workaround mentioned earlier, you can solve it. But now, when you update your XenServer version, you can’t rely on Citrix. They might just remove the required kernel modules so that ebtables wouldn’t work any more. Sounds unlikely? Well, reality is that ebtables did work until XenServer version 5.5, but in 5.6 the kernel support was removed (see last post here). To fix it, you end up downloading the XenServer SDK (which includes all the open source bits they are using) and recompile the kernel yourself.

I won’t go deeper into this subjcet, but there are several issues with bonded NICs as well. And the management interface can never be on a tagged VLAN. All those are restrictions/problems solely related to Citrix’s stuff. Linux itself lets you create any combination of bonds and VLANs on as many interfaces as you want to. Unfortunately, you need to unlearn all about Linux network configuration, because if you try applying your knowledge, XenServer will overwrite your configuration as soon as you reboot (best case) or use its API or Windows client to manage NICs/VLANs.

I could go on and on and on. There are many other quirks like being unable to shutdown a VM when for some reason it can’t attach to a VNC console (but keeps trying, although you absolutely don’t need a console to shut it down); having a “force” option for many commands, which is useless, because it doesn’t force anything; being unable to remove stale shared storage; having to work around limitations which would for example disallow you to build a pool with an i7 920 and an i7 930 server; and quite a few more, which are of minor relevance in a production environment.

Don’t get me wrong. If you dig deep enough, you will find problems in any similarly complex software. And Citrix’s XenServer is not a bad product at all. Much of the functionality like live-migration isn’t available in VMware’s free version ESXi, and said free version doesn’t run on top of CentOS but on a custom Linux, which officially you can’t access via SSH (there are ways though, but you can’t expect any support at all). Also, XenServer’s GUI is self-explanatory and easy to use — and certainly one of the main reasons for using XenServer, because whoever is going to use it after you set it up for them, they won’t have many problems getting started.

However, if you don’t have lesser knowledgeable people using it later, and if you don’t mind going the extra mile, you probably get most flexibility and reliability if you set up Xen instead (the vanilla or “real” one, not XenServer). XenServer doesn’t really provide any additional functionality, which isn’t available in Xen. (Some people even say the opposite is true, and you only get full Xen functionality if you purchase XenServer’s extra licenses; I wouldn’t go that far.) It does add convenience with its GUI and toolstack though, which you’d otherwise have to implement yourself — snapshots, shared storage use, starting up any type of guest OS etc. Most of those things aren’t exactly rocket science; only a few are a bit more tricky. But you can script/automate them as you please and you don’t need to expect any bad surprises caused by 3rd parties.

For example, I disabled Xen’s bridging code (by commenting out a single line in their scripts) and do the whole network configuration with standard OS tools, keeping it independent and consistent for future updates. (More details here.) Snapshots are easy enough to do with LVM, too. Live-migration I haven’t tested yet, but it doesn’t look too difficult to do either. (We don’t really need that feature here anyway)

What I’ve struggled with was getting different OS running, namely FreeBSD. But now that I have sorted that out, I can easily clone and fork more FreeBSD VMs on the vanilla Xen machines. Hence, Citrix XenServer isn’t providing any benefits there either.

As you can see (and as the title suggests), I’m considerably fed up with XenServer’s quirks; some of them are too huge to accept them in production environments. Consequently, we’re going to “migrate” back to Xen, where we can. (Admittedly, in some environments we won’t be able to do that for another year or so.)

Once you’ve worked out how XenServer stores VM backups (yep, they did their own thing there too, and the format is really stupid), it’s not too difficult to convert them. I’ve done that for both CentOS and FreeBSD XenServer images. They run smoothly on vanilla Xen after converting them back.

Once again the “keep it simple” motto wins. Additional toolstacks and bloat cause more problems than necessary, and the manufacturer turns out to be the only one benefitting from it — as often is the case. So long, XenServer — Hello Xen!

(Update: Only three hours after I published this, one of our XenServers started refusing to create new VMs from templates…)

(Update 2: It’s cursed. Yesterday I was all of a sudden unable to attach any block devices, hence I was unable to start new VMs, reboot existing ones, or increase storage. I’m not the only one, who faces that problem and does not get any help from the experts at Citrix.)

(Update 3, Aug 25th: Done. Last weekend we’ve transformed the last remaining XenServers to vanilla Xen. Thanks to the twin-design, this went through without any downtimes whatsoever; was a major piece of work though, but certainly worth it. Chapter closed. )

Admittedly the learning curve for both is quiet steep, and everyone’s well-advised to spend some serious time evaluating them. When I started looking into EC2, I only had a rough idea of all the services they offer. I was quite overwhelmed, how many related services EC2 (or more precisely AWS) entails:

• EC2, the cloud, which runs your instances (also known as virtual machines, Xen based) in one out of four regions (US East/West, EU, APAC) and one out of two-four availability zones in each region
• Elastic Load Balancing (ELB), giving you the opportunity to spread load across instances, obviously
• Elastic IPs, allowing you to assign (and re-assign) static IPs to instances of your choice
• Simple Storage (S3), which guarantees replication of your stored data in three different locations, enabling it to survive an outage of two entire data centres (or one data centre, if you opt-in for the “reduced redundancy” option, which is a little bit cheaper — you can choose that for every file stored individually)
• EBS (Elastic Block Storage), enabling you to create RAID-backed volumes of any size and attach them to any of your EC2 instances; on top of that you can create snapshots (which are internally stored on S3) within seconds
• RDS (Relational Database Service), basically a MySQL offering, in either single, single/hot-standby, master/slave, or master/multi-slave setups, with nodes spread across different availability zones
• CloudWatch, which entails monitoring facilities for most of the services
• CloudFront, a multi-region CDN-like service
• SimpleDB, Map/Reduce
• Route 53 DNS services (beta)
• DevPay, Flexible Payments

All these services have one thing in common: They can be managed entirely via different APIs and command line tools. There’s nothing which you can’t automate, if you spend some time and effort to actually understand how it all fits together! It’s certainly very overwhelming in the beginning, and Amazon clearly doesn’t target customers who might want to fire up one or two instances and that’s it. It’s way too complex for that. And it requires an entirely different approach, for example an instance and all its data is lost when you terminate it. And all resources are very dynamic — for most simple use-cases too dynamic (you don’t know which IP or hostname your instance will have; most provided OS images won’t suit your needs, so you’ll need to build your own). But if you are interested in creating environments for your applications, which come with both high availability and scalability, then EC2 is definitely worth a shot. Amazon gives you the bullet-proof and battle-proven infrastructure and tools — you need to decide and find a way how to use them for your requirements.

Amazon offer the AWS console for very basic management of your resources. Very basic. You’ll soon find out that it can’t offer things, which you really will need:

• creating a snapshot, which you can use to boot another instance from (or as a backup to start the same instance again, when it fails)
• setting triggers for the CloudWatch monitoring (or alarms as they call it in their API)
• bundling your instance (or parts of it) and backing up on S3
• moving instances between availability zones
• configuring the RDS MySQL server
• and many more things

All of these things can be done via API (in Java, PHP, and other languages, or via command line tools, which can all be downloaded from Amazon). Some of them are trivial, most are not. Flexibility takes its toll. Consequently, you should be prepared to spend some time tailoring your own toolset. There are some 3rd party offers out there (notably the best on is s3cmd, which allows rsync-style file transfers between instance and S3 buckets). They may or may not suit your needs.

I’ve spent the last two weeks creating my own toolset. With very simple commands I can now build fully bootable AMI images for different Linux setups both in 32bit or 64bit (EC2 instance types differ in terms of architecture!), create bootable snapshots from running instances, detect instance failure and restart from the most recent snapshot (including re-assigning the elastic IP), set tags and other information/attributes on all sorts of resource types, create volumes (empty or from snapshot) and attach them to instances, hook instances into a load balancer, read all relevant CloudWatch metrics and feed them into RRD graphs, clone instances on-the-fly, launch any number of clones, manage security groups and keypairs etc.  Basically everything the AWS console can plus a few necessary features on top of that — with a single shell command and no more than 2-3 parameters each. I’m not exactly a developer and started doing this merely as a proof of concept (but then went further than originally intended to). If I can do that, some of you bright-minded developers can do a lot better for sure

This was AWS management covered. But how about managing the actual instances (their OS internals)? What if, for example, you want to deploy a web application on four identical, load-balanced nodes?  Should I create a dedicated image for that (not too difficult with my toolset)? Or would it be better to have a look into Puppet at last? I went for the latter. I’ve got customers on my own clusters outside of EC2 (mostly based on Citrix XenServer), and that environment is growing continuously. It’s about time that I simplified management there as well.

Consequently, I decided to take my EC2 proof of concept another step further. After getting acquainted to Puppet, I’ve deployed it on a playground-style bunch of EC2 instances and told it to install/configure various things. The language structure really gave me hard times in the beginning, but once you get used to it, you can almost write it down as you think.

The next thing I wanted to achieve was that puppet connects to the puppet master as soon as the instance is started. There were some obstacles in the way, though: AWS assigns hostnames dynamically, but your puppet master would need to know that hostname in order to sign the certificate used for communication between both. Catch 22 situation. Resolved by writing a tiny web service which allows the instance to figure out and set the hostname I assigned (and dynamically added to a DNS server as well) rather than using Amazon’s one. This happens during startup just after the network interface comes up, so that all running services use the correct hostname. Puppet then takes over at the end of the first startup of the instance and installs/configures as told by the puppet master. This way you can fire up a whole cluster, hook it into the load balancer, and are ready to go live in just under three minutes. Fully automated. And the monitoring mentioned earlier would pick up metrics via CloudWatch instantly.

I’ve heard it all in theory before. However, I wanted to see my own working proof of concept for a few things (and some others, which are still in progress). I’m pretty amazed actually, how much flexibility and reliability (often a contradiction in terms) AWS offers. You just have to embrace a slightly different model of implementing things (you’ll like the term “ephemeral”, which Amazon have chosen for a reason!).

So now the next question would be: How much does it cost. Is it really saving costs as many people state? Frankly, I don’t know yet. It may do. Surely, it reduces upfront costs, as Amazon won’t charge any setup or recurring fees, unless you opt-in for their “Reserved Instance” schemes, which are actually significantly cheaper in the long run. For example, a Micro instance (640 MB Ram, 1.7GHz Xeon; the smallest instance type) would usually be charged at US$ 0.025 per hour, which equals to US$ 219 per year if running full-time. If you commit to a year, paying US$ 54 one-off, your hourly rate is reduced to US$ 0.01, which together equals to yearly costs of US$ 117.60 or less than US$ 10 per month! That’s a smashing 50% discount almost. Higher discounts possible, if you can commit to 3 years.

However, the pricing is somewhat difficult to decipher and costs impossible to predict. I don’t actually know yet, how many IOPs (I/O operations) my EBS volumes and snapshots will generate. I can’t exactly tell how much S3 storage I will use. Also, I don’t know exactly what to expect on the inter-availability-zone traffic scale. Or the RDS (MySQL) read/write operations. Surely, previous monitoring gives me very rough estimates, but not good enough to make an educated guess as to what costs to expect on EC2. I will have to keep an eye on that over the next weeks and months, and also find some tools to get all these figures from the usage reports (CSV or XML files, downloadable from Amazon). At least you can see how your usage translates into actual costs for the current billing period, updated every few hours. So the costs wouldn’t hit you as a big surprise

On the plus side, you never pay for any over-capacity, which you would need to account for, if you built everything in-house. When you build infrastructure like that on your own, there are different things, which scale more or less dynamically (if you’ve got 10 servers already, buying two more doesn’t do a harm). But you’ve also got devices where upfront costs are enormous, because you buy them from the point of view, of what you might need in the foreseeable future, not what you do need at this very moment. Storage devices are a good example. A chassis from NetApp with only a few drives cost you an arm and a leg; then you can scale it a bit for a reasonable price; and then you’ll need another one sooner or later. But you always end up paying for more than you actually use at any given point. Same for networking devices.

Clouds like EC2 take those massive entry-costs from you (and your customers), which saves painful budget discussions. They’ve got a brilliant, scalable infrastructure, and one would be bold to assume that you could build anything better at a reasonable price (also take availability in distinct and independent data centres into account!). Now that they’ve got their PCI DSS certification, one of the biggest remaining concerns (what about data security in a proprietary, shared environment?) for many customers is gone, too.

I’ll go through the other proofs of concept on my list, see how usage translates into actual costs over time, and may then be able to add some very interesting offers to my company’s portfolio. Stay tuned

Surely, the cloud is not the solution to all problems (although it’s commonly propagated as that), but with decent automation and tools it can improve or at least add value to a variety of services.

Let’s say you are running FreeBSD on a server, and you need to do a major upgrade (that is from 6.x to 7.x). This process can take ages, if your machine is not running the latest hardware, and/or you have a lot of 3rd party software installed (ports). I’m not talking about an impatient person’s definition of ages, or about the one of a customer, who claims hundreds of quid financial loss in 20 minutes downtime on Sunday morning 1:30 am.  I’m talking about ages as in many hours.

Of course, a FreeBSD upgrade doesn’t require to be offline while it’s proceeding. But you will need to reboot. And as a rule of thumb, one can assume that dependencies in the ports will break. Usually only one or two of them, but it requires manual work, and can cause an unpredictable partial downtime, which is longer than it takes to reboot the machine.

So how can virtualisation help here? In a nutshell, it allows you to do the whole upgrade on another virtual machine. You can take a snapshot of the production machine, start it as a new VM, and do your work there, while the original VM stays online.

This also reduces stress enormously, because if you break something during the upgrade, there’s no time pressure to fix it. You can spend as much time as it takes to finish your work properly. Cool, isn’t it?

And when you’ve finished your work, you can inform your customer about an upcoming 1 or 2 minutes downtime for a major system upgrade (which you have already finished).

All you need to do when the time has come, is to sync files which changed during run-time (for example mail folders), change the network settings in order to make your upgraded snapshot take over, and then you can safely decommission the old VM. It really is as easy as that.

Here’s a list of features I want to implement:

• full failover
• high security
• scalability
• load balancing
• prioritisation based on the distance to the micro data centres
• clustered filesystem across micro data centres

The idea is to put a couple of virtual machines on both of the “micro data centre boxes:”

• pfSense firewall
• Pound load-balancer
• MySQL-Cluster nodes (NDB Cluster)
• GlusterFS cluster storage nodes
• PowerDNS with geo plugin (to deliver IP resolutions depending on the source of a query)
• Apache instance(s) accessing the GlusterFS storage and the MySQL cluster

The only VM which has a public IP is the pfSense firewall. It deals with the requests and does the inter-box traffic shaping to ensure stability for the cluster filesystem (which replicates itself between the “micro data centres” in real-time) and the MySQL cluster, which is also spread over both physical boxes. The PowerDNS server resolves domain queries depending on the source of the requests. One server resides in the US, one is located in the UK (for now); so European customers will always be redirected to the UK server, whereas American customers will always see the US based server first.

The reason, why everything lives in its own VM is that choosing this approach, it is possible to easily move a VM onto a more powerful machine if necessary. The whole “micro data centre” can grow as required. But from the beginning — that is the key interest of this experiment — everything is fully redundant and it does not matter, if a single VM or one of the two boxes entirely fails. That’s much closer to 100% uptime than any single-data-centre approach (which you will most commonly find).

Actually, I have never had a look behind the curtains of a Content Delivery Network (aka CDN). But I am pretty sure that the idea is similar, although they certainly do not use VMs or only two servers. So the implementation will differ. Anyway, I’d like to see how my approach works. It’s an experiment using free software only (which includes ESXi since it was released with a new license end of July).

I started this, because I like VMware ESXi and want to stress-test it a bit. Also I like the pfSense firewall (which is FreeBSD based). And I do have good experience with GlusterFS (also not yet in an inter-continental setup). MySQL NDB is new to me. So is location-based domain resolution based on PowerDNS.

I’m pretty sure that VMware Infrastructure with the HA-AddOn would be much easier (and way more expensive!), but this is about learning new things and combining them with existing knowledge… Yes, call me a freak.

So, you now know the idea. In the next articles of this series, I will go into detail about all the things I do and how I am getting on with the project. Stay tuned.

[Added August 17th] Here’s a small diagram how I thing it could look like:

It’s a prejudice that virtualisation is only interesting for so-called Virtual Private Server providers or for big companies who need to run loads of tests for their software releases on different platforms and configurations. Also, you do not need a bunch of servers or a blade-center to take an advantage on virtualisation. In this article I’d like to give an example of what can be achieved with virtualisation apart from those typical and well-known scenarios.

Firstly, have a look at the load average of your own server(s). If it stays below 1 (per CPU/core) most of the time, you are actually not making use of all the power your machine has to offer. Hence, you are actually wasting money! You might say: “I am not yet making use of it, but my business is supposed to grow and some day I will need all of the idle resources.”

That’s absolutely fine, and 90% of all small companies will agree with your approach. However, I would like to outline what virtualisation can do for everybody who owns (or rents) a dedicated server. I’d like to show how

• you can improve access to your system for maintenance tasks
• you can use idle resources and free them when they are needed for other purposes
• virtualisation can increase stability and security of your server

First things first, virtualisation is not as complicated to set up as commonly expected. My favourite implementations are VMware Server and VMware ESXi (free since 28/07/2008), because they enable you to run any operating system within virtual machines without any changes to their kernels whatsoever. Moreover, WMware products have a long history and have proven that they are rock-solid. However, if you are sure that you will not run anything but Linux, you might also want to have a look at XEN, OpenVZ, Linux vserver or other implementations. See the previous article for an overview of available products.

The easiest installation is offered by VMware ESXi Installable. All you have to do is to insert its bootable CD (if you have a SuperMicro KVM-over-IP, a Raritan eRIC G4 card or similar, you can do that remotely as well). It will install itself onto the server, ask you some questions, and that’s it. ESXi is a hypervisor which does not require any host operating system to run on. You are ready to install any piece of operating system you want as a virtual machine through the VMware Infrastructure Client (Windows application, for free, part of ESXi).

If you don’t have KVM-over-IP or local physical access to the server, you could also ask your server provider to do that for you. It takes less than half an hour and is very easy to do.

Anyway, the subject of this article is not how to set up virtualisation. I just wanted to give an example. The topic here is, why you may want to consider using virtualisation.

How to improve access to your system for maintenance tasks

Does you server have KVM-over-IP or do you have physical access to the server? Then this might not apply to you. All others regularly generate costs when they want to do simple things as an ReiserFS or ext3 filesystem check, or when they want to compile a new kernel and it does not work out as expected: They have to ask their server providers to grant access to their servers via KVM-over-IP (if possible). With some hosts that can be kind of a nightmare!

So how does virtualisation help here? In case you are using VMware, each virtual machine can be accessed via a remote console. You can change BIOS settings, monitor the boot process and access your machine even if it does not have a SSH daemon running, as if you were sitting in front of it. The filesystem is corrupted? Just insert an ISO image of your favourite rescue CD into the virtual CDROM drive and boot from it. Do your filesystem maintenance or fix the problems with the custom kernel, eject the CD image, and boot again. There you go: Within minutes your problems can be solved — anytime you want and without any additional costs.

As for VMware you could also have a tftp server running in another virtual machine and boot VMs via PXE! That’s quite advanced but very helpful, should you ever need it.

How to use idle resources

Would be a shame to waste resources (and money!) on a machine, wouldn’t it? So why not running the main VM with most of the resources assigned, while still keeping spare resources available to install completely different things on the same machine?

Of course you can usually do that on a non-virtualised server as well. But how do you control resources of less important services and tasks? And would you really want to mix experimental stuff with your production servers? What about security in that case? Maybe you may want to test other Linux distributions? That’s all easily possible with a virtualised server.

Just tell the hypervisor to prioritize your most important VM(s) or hard-limit resources of your additional VMs. Then you will not see any impact of your experimental stuff on the production services at all! You do not need to worry about security, stability, clean un-installs of failed experiments. You will not experience a single second of downtime of your production VM while you are doing the most sophisticated experiments on another VM!

You also might to do more than just experiments on the same server. For example, you are starting with your business and want to keep costs at a minimum in the beginning. However, you expect your business to grow quickly. What you could do is to start with a single virtualised dedicated server and split it into logical units, each of them running on a separate virtual machine. As soon as you realise that the server may reach its performance limits, you simply migrate one or more of the virtual machines onto another physical virtualised server. The interesting thing here is, that you do not need to worry about the hardware it is running on. All VMs can have the same set of virtual hardware components. To the guest operating system, they all look the same, no matter what network cards or RAID controllers physically exist in a server. That makes migration quite easy, even if you do not use VMware or its migration tools.

Virtualisation can increase stability and security of your server

Okay, this one is a bit more complicated to explain. So when does a server usually become unstable?

• when it runs out of memory and swap space, so that the kernel randomly has to kill tasks/services
• when the load gets too high (often in conjunction with heavy swapping due to lack of memory)
• when it is being attacked from outside (DDoS)

The problem with these reasons is that the results are unpredictable and may lead to data loss or data inconsistency. Sometimes a hardware reset is necessary, as no remote access is possible any more (SSH daemon might have crashed already or takes ages to establish the SSH session).

As long as it’s not a DDoS attack which causes the problems, virtualisation together with it’s resource limitations for VMs and a remote console (as in VMware) can help to log onto your VM and fix the issues. In the worst case you might have to shut it down, but that’s not a hard-reset via remote power bar control (which is the worst thing you can do to a server). You just have to restart the VM, which is so much faster than restarting a physical server!

And even in case of a DDoS attack, you might be better of with virtualisation, if you have two NICs connected one of which is on a local/maintenance network. Then you still could have full access from there and could use the remote console to block ports or source IPs. That works even better, if you run a firewall within one of the VMs and have a virtual local network infrastructure set up within the virtualisation.

Now that sounds a bit complicated, doesn’t it? You might want to read about my example setup, which is not very complicated but very effective.

My example setup

To start playing with the free VMware ESXi, I ordered a nice machine from SoftLayer which comes with KVM-over-IP (and allows remote CD image mounts). So I literally do have full control over the server. But you could also choose any other host and do not need a KVM at all, if the host agrees to install VMware ESXi for you (which is easy and does not take more than half an hour).

After successful installation, I used the Virtual Infrastructure Client to set up a couple of virtual switches:

• vSwitch 0 is connected to the public interface card
• vSwitch 1 is a host-only switch for local networking between the VMs (not accessible from outside)
• vSwitch 2 is connected to the interface card on the management network (SoftLayer provides access to this special infrastructure via VPN — very nice!)

My first VM was meant to become the firewall. Now, you may ask: “Why the hell do you put the firewall onto a VM? You could use iptables, pf, ipfw within the VM.” Sure, I could. But why should I want to maintain firewalls for each single VM when I can do that centrally? Moreover, why should I reinvent the wheel? There are plenty of good firewall solutions out there, which come with so many extra features out of the box.

I went for pfSense. It comes with literally everything you might want to implement:

• stateful firewall
• NAT port-forwarding and 1:1 NAT (interesting if your dedicated machine has multiple IPs)
• web interface
• different VPN services (PPTP, IPSec, OpenVPN)
• traffic shaping (queues, prioritisation etc.)
• bandwith monitoring
• netflow hooks
• SNMP
• DHCP/DNS
• and a bunch of other packages which can easily be added to the configuration

The pfSense VM connects exclusively to vSwitch0 (public network) and to vSwitch1 (host-only local network). That means, all traffic to the other VMs goes through this firewall. There’s no way to circumvent that — neither for public sources nor for the VMs, which are only connected to vSwitch1 locally and listening on private IP ranges (192.168.0.0/16, 10.0.0.0/8, 172.16.0.0/12). The VMs get their local IPs assigned by  pfSense’s DHCP (nice for quick experimental VM setups) and have to use pfSense also as gateway and DNS.

Currently I am experimenting with TFTP to boot VMs via the network/PXE, which is possible with VMware ESXi and VMware Server. That will allow extremely quick and slim VM installations.

Thanks to the many features of VMware ESXi, you can literally set up a virtual data centre in a box. Ok, a small one. And you gain full control about everything which happens in your small data centre.

So, to cut a long story short: Virtualisation helps to

• increase control over and maintainability of your services
• centralise certain tasks (like the firewall or bandwith control)
• make use of spare resources and reduce costs
• ease migration in case the hardware is no longer sufficient (you don’t need to worry about the hardware platform the VMs are running on)
• speed up development/experiments as a VM is deployed much quicker than a dedicated box

I hope, this article helped at least a bit to make you curious about virtualisation. Personally, I think that virtualisation does not only target big companies or VPS hosts. It is interesting for everybody who owns/rents dedicated boxes. And it is certainly worth the effort to give it a go!

This article is about virtualisation. Caution: Long article!

As I am professionally dealing with all sorts of Linux flavours (Debian/Ubuntu, RedHat, CentOS, Fedora) and BSD/Unix derivates (Solaris, FreeBSD, OpenBSD), I have always been keen on a real virtualisation solution. I have come across many different (para-)virtualisation solutions in desktop and server environments over the last couple of years. Just to give an outline, I will now list all of them (without any particular order). I’d like to give an overview of the main differences later. So here’s the list for virtualisation approaches:

• VMware ESXi / VMware Server
• XEN
• OpenVZ (Virtuozzo open source fork)
• Sun xVM (for some reason, many people do not see this one)
• QEMU (honestly, who of you knows that one? )
• FreeBSD jails
• Linux vserver
• VMware Fusion for Mac
• Parallels Desktop for Mac
• Linux Virtual Server (LVS)

Okay, here we have a bunch of different things in one big pot… First, we have a couple of desktop based products (Parallels Desktop, VMware Fusion). I am not going to go into details about them. Personally I think that for professional testing/dev environments, VMware Fusion is slightly better than Parallels Desktop, because its networking components are more sophisticated and also allow to boot via PXE/tftp from within a virtual machine! On the other hand Parallels Desktop is said to be slightly ahead regarding graphics performance. I did not see much difference. Apart from that, both do pretty much the same and allow to run a variety of operating systems on top of the MacOS platform. I am using both heavily and have never had any real problems (tested with Windows XP, FreeBSD, Solaris and a bunch of Linuxes as guests). Okay, enough about Desktop virtualisation, especially as neither VMware Fusion nor Parallels Desktop are for free. Well, they are almost and definitely worth the money. Should you ever need to run different OS on your Mac, I can recommend both.

Sun’s xVM sits somewhere in between desktop and server virtualisation. The Mac implementation is not yet very mature, so it is out of question for me. But on Linux it is known to run quite well. Certainly better than the Wine patchwork thingy.

And here we go. Finally, eh? How to virtualise your server for developement, testing, or production use…

The rising star and increasingly popular is certainly XEN. I used to work with it way back in the past (early 2.x, if I remember correctly). That time, it was hardly more than an experimental virtualisation. It was great to use for running a couple of Linux instances on one machine. But you could not expect much more from it. Quite annoying was the fact that you had to patch the host and guest kernels. Also, it was anything but well documented. As it is becoming more and more popular, I would expect that it has become more mature over time. For sure I can tell that it performed quite well. When it comes to more than “just” Linux guests, XEN does not have much to offer. Support for Windows seems to become better, but FreeBSD or Solaris are still out of question.

If I needed to go for a para-virtualisation which allows to run Linux guests only, I would go for OpenVZ. It is documented very well and offers loads of opportunities to assign ressources to its guests. As an Open Source fork of Virtuozzo, it moreover has proven reliability very well. It comes with pre-compiled Linux kernels and a couple of tools. All of them are easy to install and work right out of the box. As from my experience, the performance is awesome.

Neither XEN nor OpenVZ do have their own graphical interface. That’s where a tool named HyperVM kicks in. It’s not for free but claims to be the only graphical front-end for both XEN and OpenVZ. I’ve never tried it, but people who fancy graphical tools rather than command line tweaking, might want to give it a go.

Other Linux-only virtualisation approaches are Linux Virtual Server and Linux vServer. I’ve tested both in an early stage, but they really did not convince me. None of them was capable of assigning resources to guests (resources in terms of at least RAM and CPU soft and hard limits). OpenVZ does that much better, as it enables you to limit almost everything. However, Linux vServer at least runs very stable (we used that on Ubuntu Server production machines in the last company I worked for).

A bit of a special role takes QEMU. Unlike the other mentioned virtualisation approaches, it does not need any kernel manipulation or sophisticated configuration. It’s merely a userland program which provides an environment for a single virtual computer. This is great if you quickly want to test something on another platform. “Quickly” in terms of easy to set up, not in terms of performance. Compared to kernel-level virtualisations it performs rather poorly. Where that isn’t important, it however is a great tool.
I used it to prepare FreeBSD images for a server which I did not have KVM access to. After preparing the images on my local client, I uploaded them to the rescue system on the server (booted via PXE) and then wrote the image directly to disk. Those days, there was no other way for me to do it; and it worked unexpectedly great!

Another special approach can be found on FreeBSD: so-called jails. It is pretty similar to Linux-only virtualisation on Linux systems, although it’s obviously for FreeBSD only. You get fully separated guests (jails), which have to get an IP address assigned (kind of bridged mode) and operate completely independently from the host system. Together with layered file-systems, read-only and null mounts, you can set up very slim but secure guests. Also, it is possible to run guests on higher secure-levels (a FreeBSD thing), and make them extraordinary safe. They perform incredibly well.

And here we go: VMware Server and ESXi Installable. It’s not a secret that I like them more than any other virtualisation approach, because they allow to run any imaginable operating system as guests. Moreover they offer extremely good tools to create and manage guests and to overlook the health and performance. With the VMware Converter, it is possible to transfer guests between different dedicated servers. All of this for free!

When ESXi 3.5 was released for free on Monday, July 28th, I decided to install it immediately on a private server, because I wanted to find out in which way it differs from VMware Server (which has been for free since its first release).
First of all, it does not depend on the host operating system, because it is just a hypervisor which then allows to install any operating systems as guests. So basically, we do not have a host operating system at all here, which certainly improves stability, performance and maybe also security. Secondly, it allows to set up a virtual network infrastructure as well. Virtual switches, VLANs,… Basically it means, that ESXi allows you to run a small data centre virtually. This of course offers great opportunities for a variety of testing scenarios.

Apart from that, it does not really differ from VMware Server. Both ESXi and Server can be maintained either through Perl scripts (remotely) or through the VMware Infrastructure Client (also free, but requires Windows to run). It’s needless to say that VMware products are rock-solid and reliable due to the fact that they are widely spread and have been available for many years. Only the EULA of VMware ESXi and Server is still a bit mistakebly. I started a discussion thread in the VMware community forum to get that clarified.

In another blog entry, I’d like to explain why virtualisation is much more interesting for many people than they might think. And I’ll also give an example with my own private server setup.