So where’s the but? Here it comes: …but if something unexpected happens, you are seriously screwed. Here are a few examples from the past couple of months.

Changing a NIC, which is also management interface, of a pool server — This was about the worst nightmare I’ve ever had. What you’d expect to do is: shutdown the machine, open it, replace the NIC, close it, switch it on again, wait for it to boot and start the VMs, done. What really happened is: I had to actually wipe and re-install the whole box, because there was apparently no documented, reverse-engineerable, or otherwise known way to just simply change the MAC address somewhere, because that is managed by the pool master. Now, as the NIC was broken, the master wasn’t able to communicate with the pool server any more (not even on the second NIC, because that was not the management interface). Attempts to change it failed. Not even the “xe” tool was functional any more, so I couldn’t really gather the UUIDs in order to search through configurations etc. The master refused to talk to the pool server, and the pool server with the broken (and afterwards replaced NIC) refused to let me change anything, because that should be done on the master. Catch 22.

I consulted the official support forum, but nobody knew an answer there either. I’m sure there is a way to change it easily. After all it’s a Linux box with a modified Xen, but still not an unaccessible blackbox. Hang on… actually it felt a bit like that. I would like to think that Citrix certainly knows an easy solution, but as I’m not paying thousands of Pounds for a product, which is almost entirely based on free software, they of course kept quiet. (The bloody toolstack, which complicated things, is their own development, by the way.)

End of that experience was that I had to remove the server from the pool (XenServer would then wipe the box, so you can’t re-join the pool later, either… awesome). After a clean setup and restoring all the VMs from previously created snapshots, the machine was finally able to join the pool. That was 6 hours after the NIC broke. Fortunately all VMs have an identical twin running on another machine, so it didn’t cause downtime (except a few minor hiccups while I was fiddling about with network settings). Otherwise all websites/applications would have been offline for 6 hours.

Without the XenServer toolstack, I could have resolved the issue within 10 minutes, which includes all of the steps mentioned earlier (what I would have expected).

I learned my lesson from it. As live-migration of VMs isn’t really necessary in most cases (my customers’ applications don’t benefit from it), it’s actually better to not form pools of your servers. Disconnected standalone servers are a lot easier to maintain and you don’t risk side-effects with pool members, because there aren’t any. The only real downside is that VLANs need to be configured individually on each server. Same applies to shared resources (NAS etc). But that’s fine.

Another almost unbelievable example is deleting snapshots. I create them all the time, because if something goes wrong, or someone breaks a VM setup, you want to be able to roll back to a previous version. Snapshots are one of the biggest advantages of virtualisation. A whole VM can be brought back to an older state within seconds. Or you can export it and reimport it elsewhere, clone another instance from it, work there, swing later. Anyways, if you use that feature often, it fills your disk (even the huge disks you get nowadays). So you regularly delete them and get your space back. Right? Nope, wrong. With XenServer you may or may not get your space back. When your monitoring tells you that you are running out of disk space, although you haven’t done anything but rotating snapshots in a while, you scratch your head in disbelief. Well, at least I did. Unfortunately, the official documentation confirms my observations. When I first read that reclaiming space causes downtime, I wasn’t sure if laughing or crying was the best course of action.

In a production environment, you can’t just go ahead and suspend VMs just to get space back. Even if you only reduce performance (without causing downtimes, as we’re running twins of everything), you need to make affected customers aware of it. And how do you explain that? “Sorry, Sir, I need to suspend your service, because I need to delete old snapshots.” They’ll think you’re taking the piss.

Again, this “feature” is brought to you by Citrix’s toolstack, not Xen. If I decide to delete an LVM-based snapshot of a running VM on Xen, I can do that any time. No need to suspend anything or to manually reclaim free space afterwards.

My favourite subject is VLANs. I don’t know how many hours I’ve wasted trying to find what I did wrong, just to figure out in the end that it was not my fault… Citrix apparently manipulated the bridge code and never really tested it. You have to actually install ebtables (iptables for bridges, if you will) to work around that issue. I observed exactly the same thing as the poster there, and many others did, too. Their forums are full of problems related to VLANs and NIC bonding. Problems get worse with two NICs. VLANs may work out-of-the-box on both, only one, or none of the NICs. Apparently it depends on the used NIC (well, I’m assuming here that nobody uses old NICs without VLAN support any more nowadays), which of the NIC is management interface, and a couple of other factors like weather, mood etc.

Once you know about the workaround mentioned earlier, you can solve it. But now, when you update your XenServer version, you can’t rely on Citrix. They might just remove the required kernel modules so that ebtables wouldn’t work any more. Sounds unlikely? Well, reality is that ebtables did work until XenServer version 5.5, but in 5.6 the kernel support was removed (see last post here). To fix it, you end up downloading the XenServer SDK (which includes all the open source bits they are using) and recompile the kernel yourself.

I won’t go deeper into this subjcet, but there are several issues with bonded NICs as well. And the management interface can never be on a tagged VLAN. All those are restrictions/problems solely related to Citrix’s stuff. Linux itself lets you create any combination of bonds and VLANs on as many interfaces as you want to. Unfortunately, you need to unlearn all about Linux network configuration, because if you try applying your knowledge, XenServer will overwrite your configuration as soon as you reboot (best case) or use its API or Windows client to manage NICs/VLANs.

I could go on and on and on. There are many other quirks like being unable to shutdown a VM when for some reason it can’t attach to a VNC console (but keeps trying, although you absolutely don’t need a console to shut it down); having a “force” option for many commands, which is useless, because it doesn’t force anything; being unable to remove stale shared storage; having to work around limitations which would for example disallow you to build a pool with an i7 920 and an i7 930 server; and quite a few more, which are of minor relevance in a production environment.

Don’t get me wrong. If you dig deep enough, you will find problems in any similarly complex software. And Citrix’s XenServer is not a bad product at all. Much of the functionality like live-migration isn’t available in VMware’s free version ESXi, and said free version doesn’t run on top of CentOS but on a custom Linux, which officially you can’t access via SSH (there are ways though, but you can’t expect any support at all). Also, XenServer’s GUI is self-explanatory and easy to use — and certainly one of the main reasons for using XenServer, because whoever is going to use it after you set it up for them, they won’t have many problems getting started.

However, if you don’t have lesser knowledgeable people using it later, and if you don’t mind going the extra mile, you probably get most flexibility and reliability if you set up Xen instead (the vanilla or “real” one, not XenServer). XenServer doesn’t really provide any additional functionality, which isn’t available in Xen. (Some people even say the opposite is true, and you only get full Xen functionality if you purchase XenServer’s extra licenses; I wouldn’t go that far.) It does add convenience with its GUI and toolstack though, which you’d otherwise have to implement yourself — snapshots, shared storage use, starting up any type of guest OS etc. Most of those things aren’t exactly rocket science; only a few are a bit more tricky. But you can script/automate them as you please and you don’t need to expect any bad surprises caused by 3rd parties.

For example, I disabled Xen’s bridging code (by commenting out a single line in their scripts) and do the whole network configuration with standard OS tools, keeping it independent and consistent for future updates. (More details here.) Snapshots are easy enough to do with LVM, too. Live-migration I haven’t tested yet, but it doesn’t look too difficult to do either. (We don’t really need that feature here anyway)

What I’ve struggled with was getting different OS running, namely FreeBSD. But now that I have sorted that out, I can easily clone and fork more FreeBSD VMs on the vanilla Xen machines. Hence, Citrix XenServer isn’t providing any benefits there either.

As you can see (and as the title suggests), I’m considerably fed up with XenServer’s quirks; some of them are too huge to accept them in production environments. Consequently, we’re going to “migrate” back to Xen, where we can. (Admittedly, in some environments we won’t be able to do that for another year or so.)

Once you’ve worked out how XenServer stores VM backups (yep, they did their own thing there too, and the format is really stupid), it’s not too difficult to convert them. I’ve done that for both CentOS and FreeBSD XenServer images. They run smoothly on vanilla Xen after converting them back.

Once again the “keep it simple” motto wins. Additional toolstacks and bloat cause more problems than necessary, and the manufacturer turns out to be the only one benefitting from it — as often is the case. So long, XenServer — Hello Xen!

(Update: Only three hours after I published this, one of our XenServers started refusing to create new VMs from templates…)

(Update 2: It’s cursed. Yesterday I was all of a sudden unable to attach any block devices, hence I was unable to start new VMs, reboot existing ones, or increase storage. I’m not the only one, who faces that problem and does not get any help from the experts at Citrix.)

(Update 3, Aug 25th: Done. Last weekend we’ve transformed the last remaining XenServers to vanilla Xen. Thanks to the twin-design, this went through without any downtimes whatsoever; was a major piece of work though, but certainly worth it. Chapter closed. )

Admittedly the learning curve for both is quiet steep, and everyone’s well-advised to spend some serious time evaluating them. When I started looking into EC2, I only had a rough idea of all the services they offer. I was quite overwhelmed, how many related services EC2 (or more precisely AWS) entails:

• EC2, the cloud, which runs your instances (also known as virtual machines, Xen based) in one out of four regions (US East/West, EU, APAC) and one out of two-four availability zones in each region
• Elastic Load Balancing (ELB), giving you the opportunity to spread load across instances, obviously
• Elastic IPs, allowing you to assign (and re-assign) static IPs to instances of your choice
• Simple Storage (S3), which guarantees replication of your stored data in three different locations, enabling it to survive an outage of two entire data centres (or one data centre, if you opt-in for the “reduced redundancy” option, which is a little bit cheaper — you can choose that for every file stored individually)
• EBS (Elastic Block Storage), enabling you to create RAID-backed volumes of any size and attach them to any of your EC2 instances; on top of that you can create snapshots (which are internally stored on S3) within seconds
• RDS (Relational Database Service), basically a MySQL offering, in either single, single/hot-standby, master/slave, or master/multi-slave setups, with nodes spread across different availability zones
• CloudWatch, which entails monitoring facilities for most of the services
• CloudFront, a multi-region CDN-like service
• SimpleDB, Map/Reduce
• Route 53 DNS services (beta)
• DevPay, Flexible Payments

All these services have one thing in common: They can be managed entirely via different APIs and command line tools. There’s nothing which you can’t automate, if you spend some time and effort to actually understand how it all fits together! It’s certainly very overwhelming in the beginning, and Amazon clearly doesn’t target customers who might want to fire up one or two instances and that’s it. It’s way too complex for that. And it requires an entirely different approach, for example an instance and all its data is lost when you terminate it. And all resources are very dynamic — for most simple use-cases too dynamic (you don’t know which IP or hostname your instance will have; most provided OS images won’t suit your needs, so you’ll need to build your own). But if you are interested in creating environments for your applications, which come with both high availability and scalability, then EC2 is definitely worth a shot. Amazon gives you the bullet-proof and battle-proven infrastructure and tools — you need to decide and find a way how to use them for your requirements.

Amazon offer the AWS console for very basic management of your resources. Very basic. You’ll soon find out that it can’t offer things, which you really will need:

• creating a snapshot, which you can use to boot another instance from (or as a backup to start the same instance again, when it fails)
• setting triggers for the CloudWatch monitoring (or alarms as they call it in their API)
• bundling your instance (or parts of it) and backing up on S3
• moving instances between availability zones
• configuring the RDS MySQL server
• and many more things

All of these things can be done via API (in Java, PHP, and other languages, or via command line tools, which can all be downloaded from Amazon). Some of them are trivial, most are not. Flexibility takes its toll. Consequently, you should be prepared to spend some time tailoring your own toolset. There are some 3rd party offers out there (notably the best on is s3cmd, which allows rsync-style file transfers between instance and S3 buckets). They may or may not suit your needs.

I’ve spent the last two weeks creating my own toolset. With very simple commands I can now build fully bootable AMI images for different Linux setups both in 32bit or 64bit (EC2 instance types differ in terms of architecture!), create bootable snapshots from running instances, detect instance failure and restart from the most recent snapshot (including re-assigning the elastic IP), set tags and other information/attributes on all sorts of resource types, create volumes (empty or from snapshot) and attach them to instances, hook instances into a load balancer, read all relevant CloudWatch metrics and feed them into RRD graphs, clone instances on-the-fly, launch any number of clones, manage security groups and keypairs etc.  Basically everything the AWS console can plus a few necessary features on top of that — with a single shell command and no more than 2-3 parameters each. I’m not exactly a developer and started doing this merely as a proof of concept (but then went further than originally intended to). If I can do that, some of you bright-minded developers can do a lot better for sure

This was AWS management covered. But how about managing the actual instances (their OS internals)? What if, for example, you want to deploy a web application on four identical, load-balanced nodes?  Should I create a dedicated image for that (not too difficult with my toolset)? Or would it be better to have a look into Puppet at last? I went for the latter. I’ve got customers on my own clusters outside of EC2 (mostly based on Citrix XenServer), and that environment is growing continuously. It’s about time that I simplified management there as well.

Consequently, I decided to take my EC2 proof of concept another step further. After getting acquainted to Puppet, I’ve deployed it on a playground-style bunch of EC2 instances and told it to install/configure various things. The language structure really gave me hard times in the beginning, but once you get used to it, you can almost write it down as you think.

The next thing I wanted to achieve was that puppet connects to the puppet master as soon as the instance is started. There were some obstacles in the way, though: AWS assigns hostnames dynamically, but your puppet master would need to know that hostname in order to sign the certificate used for communication between both. Catch 22 situation. Resolved by writing a tiny web service which allows the instance to figure out and set the hostname I assigned (and dynamically added to a DNS server as well) rather than using Amazon’s one. This happens during startup just after the network interface comes up, so that all running services use the correct hostname. Puppet then takes over at the end of the first startup of the instance and installs/configures as told by the puppet master. This way you can fire up a whole cluster, hook it into the load balancer, and are ready to go live in just under three minutes. Fully automated. And the monitoring mentioned earlier would pick up metrics via CloudWatch instantly.

I’ve heard it all in theory before. However, I wanted to see my own working proof of concept for a few things (and some others, which are still in progress). I’m pretty amazed actually, how much flexibility and reliability (often a contradiction in terms) AWS offers. You just have to embrace a slightly different model of implementing things (you’ll like the term “ephemeral”, which Amazon have chosen for a reason!).

So now the next question would be: How much does it cost. Is it really saving costs as many people state? Frankly, I don’t know yet. It may do. Surely, it reduces upfront costs, as Amazon won’t charge any setup or recurring fees, unless you opt-in for their “Reserved Instance” schemes, which are actually significantly cheaper in the long run. For example, a Micro instance (640 MB Ram, 1.7GHz Xeon; the smallest instance type) would usually be charged at US$ 0.025 per hour, which equals to US$ 219 per year if running full-time. If you commit to a year, paying US$ 54 one-off, your hourly rate is reduced to US$ 0.01, which together equals to yearly costs of US$ 117.60 or less than US$ 10 per month! That’s a smashing 50% discount almost. Higher discounts possible, if you can commit to 3 years.

However, the pricing is somewhat difficult to decipher and costs impossible to predict. I don’t actually know yet, how many IOPs (I/O operations) my EBS volumes and snapshots will generate. I can’t exactly tell how much S3 storage I will use. Also, I don’t know exactly what to expect on the inter-availability-zone traffic scale. Or the RDS (MySQL) read/write operations. Surely, previous monitoring gives me very rough estimates, but not good enough to make an educated guess as to what costs to expect on EC2. I will have to keep an eye on that over the next weeks and months, and also find some tools to get all these figures from the usage reports (CSV or XML files, downloadable from Amazon). At least you can see how your usage translates into actual costs for the current billing period, updated every few hours. So the costs wouldn’t hit you as a big surprise

On the plus side, you never pay for any over-capacity, which you would need to account for, if you built everything in-house. When you build infrastructure like that on your own, there are different things, which scale more or less dynamically (if you’ve got 10 servers already, buying two more doesn’t do a harm). But you’ve also got devices where upfront costs are enormous, because you buy them from the point of view, of what you might need in the foreseeable future, not what you do need at this very moment. Storage devices are a good example. A chassis from NetApp with only a few drives cost you an arm and a leg; then you can scale it a bit for a reasonable price; and then you’ll need another one sooner or later. But you always end up paying for more than you actually use at any given point. Same for networking devices.

Clouds like EC2 take those massive entry-costs from you (and your customers), which saves painful budget discussions. They’ve got a brilliant, scalable infrastructure, and one would be bold to assume that you could build anything better at a reasonable price (also take availability in distinct and independent data centres into account!). Now that they’ve got their PCI DSS certification, one of the biggest remaining concerns (what about data security in a proprietary, shared environment?) for many customers is gone, too.

I’ll go through the other proofs of concept on my list, see how usage translates into actual costs over time, and may then be able to add some very interesting offers to my company’s portfolio. Stay tuned

Surely, the cloud is not the solution to all problems (although it’s commonly propagated as that), but with decent automation and tools it can improve or at least add value to a variety of services.

Let’s say you are running FreeBSD on a server, and you need to do a major upgrade (that is from 6.x to 7.x). This process can take ages, if your machine is not running the latest hardware, and/or you have a lot of 3rd party software installed (ports). I’m not talking about an impatient person’s definition of ages, or about the one of a customer, who claims hundreds of quid financial loss in 20 minutes downtime on Sunday morning 1:30 am.  I’m talking about ages as in many hours.

Of course, a FreeBSD upgrade doesn’t require to be offline while it’s proceeding. But you will need to reboot. And as a rule of thumb, one can assume that dependencies in the ports will break. Usually only one or two of them, but it requires manual work, and can cause an unpredictable partial downtime, which is longer than it takes to reboot the machine.

So how can virtualisation help here? In a nutshell, it allows you to do the whole upgrade on another virtual machine. You can take a snapshot of the production machine, start it as a new VM, and do your work there, while the original VM stays online.

This also reduces stress enormously, because if you break something during the upgrade, there’s no time pressure to fix it. You can spend as much time as it takes to finish your work properly. Cool, isn’t it?

And when you’ve finished your work, you can inform your customer about an upcoming 1 or 2 minutes downtime for a major system upgrade (which you have already finished).

All you need to do when the time has come, is to sync files which changed during run-time (for example mail folders), change the network settings in order to make your upgraded snapshot take over, and then you can safely decommission the old VM. It really is as easy as that.