You, as an IT person, certainly have been asked (or asked yourself) this question in the recent past: “Should we move services to ‘The’ Cloud?”
That’s where the challenge starts. From a business person’s perspective, on a non-technical level, “The” Cloud appears to be a fixed term, maybe even a one-size-fits-it-all solution. But from an IT person’s point of view it’s a different story. Just like real clouds, “The” Cloud comes in all shapes and forms, from fluffy and bright, to dark, threatening and soon bursting into thunder and rain. “The” Cloud doesn’t exist — each and every hosting provider implements it differently and sets their own focus. There are very few common denominators, frankly. Whether or not it makes sense for your business or project pretty much depends on what you want to do, what you already have in place, and what you actually need. One of the biggest benefits is certainly that you don’t need to worry much about the future. You don’t need to buy expensive kit and infrastructure because you might require it soon-ish. That reduces overhead costs.

When I think of clouds, I think of something very versatile or flexible. In IT terms it means that I can build my own infrastructure with virtualised components, in any way I see fit. I would be able to run any operating system of my choice. I would run a variety of different VM types, built to their purpose, arrange internal networking between them, group and secure them with firewalls. Clouds move, so would my virtualised resources. I could add or remove them as I go, make them appear and disappear when and where I need them. What I created in one place could be cloned and deployed in a different place in the world. I could multiply my (pre-configured) resources if required, and remove unnecessary pieces to reduce overhead and costs. As with real clouds, this would happen on-the-fly, without disruption of service at any time.

Let’s get more technical and derive some key requirements from that. Your mileage may vary, obviously, but for me personally “The” Cloud has to compete with what I can easily build myself. I wouldn’t dare calling the ~20 servers a cloud, which I run for one client. Although most of them function as Xen hosts, and no single points of failure exist. The majority of “cloud hosting” offers out there should not call it that, either, to be quite honest with you. Three years ago they would have titled their offers “virtual private server hosting” (or VPS), but nowadays cloud and IaaS sell so much better!

Anyhow, back to the technical requirements. From a cloud offer I expect (in no particular order):

• Zero single points of failure (or full redundancy in other words), both for infrastructure/connectivity and storage
• Ability to run any operating system or flavour, both from standard and customised images
• Instant provisioning and availability of resources (if I decide to start a VM because I need it to scale out an application, I cannot wait 15 minutes or more)
• Near-realtime snapshots of storage volumes and instant restoration of volumes and virtual machines
• Very few limitations in terms of size and number of virtual machines
• Scalability (horizontally and vertically, that is up/downscaling of individual virtual machines, the number of virtual machines running a clone, clever load balancing)
• Security (VLANs between VMs which aren’t shared with others, ability to create firewalls outside of VMs)
• Billing on an hourly basis for any resource type (on-demand does not work with monthly billing cycles)
• Ability to reduce costs by committing/subscribing in advance (on a per-VM basis or as overall resource usage)
• API and/or command line tools (if you cannot automate, it’s not flexible enough)
• Excellent connectivity to main carriers and exchanges in the area
• Knowledgeable support staff (I’d rather wait a couple of hours for a helpful answer than getting by-the-book answers from underpaid live chat staff)
• Static IP addresses available, including reverse DNS records (vital for email servers)

Ideally, there were some neat extra features, which I’d appreciate much:

• Storage outside of VMs which can be used to upload backups, share or distribute static content (CDN)
• Up/down scaling of VMs (like adding RAM or horse power, attaching additional volumes etc)
• VNC access (helpful with from-scratch customisation of VM images)
• Specialised VM types/resources for high I/O workloads like databases (local SSD would be perfect)
• Encrypted storage so that data at rest is secure (you can implement that yourself, I know, but it costs your CPU power)
• PCI-DSS compliant infrastructure
• Cluster-aware volumes which can be mounted to multiple VMs (using GFS, OCFS2, CLVM and the like inside)

I know, this is a fairly massive list of requirements, but if only a fraction of them is met, it’s not a cloud. I need high flexibility and lots of options, because otherwise I could as well build it myself and likely end up cheaper. Well, truth is, the majority of these requirements are met in my own infrastructure, but if the client suddenly starts thinking about moving to “The” Cloud, I need to take all the things into consideration which we currently have. And certainly I’m not an isolated case. Surely many of you have been there, too. (I don’t even want to start the discussion whether the cloud is cheaper than running your own infrastructure. I believe in most cases it’s not, unless you tailor your applications for the particular cloud you decided to use, and make sure they allocate and release resources as and when they are really required. Depending on the application, development of such a fine-grained management mechanism can incur tremendous costs too, in particular if you are migrating rather than building from scratch.)

So, what have we got out there? What can the different clouds offer?

As you probably gathered, the discussion of possibly moving to “The” Cloud is taking place as I write this. My client came up with a bunch of cloud offers, most of which don’t get anywhere near the requirements. But in their defence, those offers did mention all the buzz words which business folks fall for. And at the end of the day it’s the business folks who shoulder the bill and the IT crowd who have to deal with whatever they get.

That’s of course a highly unsatisfying situation. What I decided to do is to look into different offers from different providers, which I selected myself. To be able to point out the differences and make the client aware of the pitfalls and how that can effect their considerations.

Only those providers which offer at least one European data centre location can be taken into consideration, due to company policies and data protection requirements. My selection includes: Amazon, Rackspace, Softlayer, CloudSigma and ElasticHosts. The list might grow over the coming days (though, for the purpose of this blog article it’s not relevant). The purpose of this article is not to give any recommendations. Instead, I want to support my thesis that no single cloud equals any other, and that therefore there’s no such thing as “The” Cloud. There are many clouds. Entirely different clouds. All clouds do have these things in common:

• They are virtualised (using different Hypervisors and different levels of vitualisation [HVM vs. PV])
• It will take considerable effort to migrate our applications there
• If we take our virtualised infrastructure and put it there, we cannot save costs at all, unless we redesign many parts of it (and if we did that on the current infrastructure, we’d get away cheaper, too)

Sadly, that’s it already. Let’s go through some pros and cons of the selected providers. I only want to highlight a few things. Don’t take this as an all-inclusive or complete feature comparison. It’s not. If you feel that I should add certain things, or if your experience differs from mine, please do use the comment option.

Amazon, with data centres in Dublin/Ireland, which I’ve been customer with for about two years, covers almost all of my must-have items. They allow me to run almost any operating system (my favourite, FreeBSD with some limitations, but it’s possible), are most flexible in terms of snapshots (near real-time), volume management, and security (security groups and firewalls). Also they allow running entirely encapsulated Virtual Private Clouds (VPCs), which can even be connected to other data centres using IPsec tunnels. When it comes to infrastructure, they are obviously ahead of everyone else, with data centres all over the globe. They do offer VM-independent storage (S3, EBS). It’s probably reasonable to call them the inventor of the cloud. Or at least they were the first to bring it to a wide audience.
In terms of billing you can save money by committing to so-called Reserved Instances, which can significantly reduce your costs by paying a one-off fee for one or three years. However, Reserved Instances are tied to Availability Zones (specific data centre in a given region), a certain instance type, and even distinguish between EC2 and VPC instances. You want to think twice which resources you will need before committing so specifically. That said, a Reserved Instance is a slot, which allows you to run a particular instance type at reduced hourly costs, not an actual instance in itself. You’re still flexible, sort of.
In terms of instance types, they’ve got a wide range from “Micro” to “Cluster Compute” and even “GPU” number-crunchers. However, there are some gaps between the instance types, and I bet many people wish for one or more instance types to fill those. Also there are some instance types which require a 32bit OS, whereas all others can run (or require) 64bit. This can be a problem when scaling vertically.
A big plus for people designing seriously large applications are the different specialised instance types for MySQL, Oracle, Map/Reduce, DynamoDB and whatever else they’ve got. Memcache instances (ElastiCache) are a very unique offer as well.
You usually get a reply to support inquiries within a day or two, if you don’t buy premium support (which guarantees response times on different levels). Alternatively you can post your issue into their forums, which most customers seem to do, and can be lucky to get a very quick response. That said, you don’t need their support often. Everything is automated (web console or API), and unless something is technically wrong (as in broken), you can always work around it. Also, there is a vast number of tools available which allows you to migrate to or from Amazon, up to the point where they let you bundle your VM or physical server, throw it onto S3, and create a running instance in EC2 from it.
Amazon is the only provider in this article, which allows you to span applications across different data centres and provide actual physical resilience on all levels. (Yes they too do have their intermittent issues, but who doesn’t? If they can’t do it, there aren’t many who can.)

Rackspace, with data centre in London/UK, have positioned themselves as competition to Amazon. However, I don’t think they live up to that expectation. What you can say is that smaller instances seem to deliver a constant level of performance, whereas with Amazon the Micro instances can be heavily throttled. However, the choice of instance types is very limited. If you want more RAM or CPU or disk space, you automatically have to increase everything else too. Hidden in their knowledge base you can further learn that bandwidth is throttled depending on the instance size as well. What you cannot get is static IP addresses.
The choice of operating systems is very limited. Snapshots cannot be taken of a running instance (I know the risks, but it’s good to have that option anyway). You can make clones of your instance, but you cannot mount the volumes elsewhere to manipulate them. And talking of volumes, well, there aren’t any. You get a set amount of space depending on the instance size. That’s it. External firewalls don’t exist, and the internal network interfaces share one network (not a private VLAN). There’s no way you can import or export VM images. Customization is not their focus. FreeBSD is not supported at all as yet. Allegedly that will change soon. It remains to be seen.
On the good side: Their support is brilliant and fast, and their user interface and API are intuitive and well documented. Rackspace’s Load Balancers allow using external backends, which can be handy, although there won’t be too many use cases for that.
While priced reasonably and performing well, Rackspace’s focus clearly isn’t on flexibility and/or anything non-standard. Personally I find their options very restrictive and almost disappointing.

SoftLayer have opened a data centre in Amsterdam/Netherlands only recently. However, I’ve been dedicated server customer with them in the past. I knew that their network quality and support are almost unrivaled. It’s one of the few providers out there which manage to make customers feel important and listened to. Considering their size and growth rate that is quite an achievement.
SoftLayer is immensely flexible with instance types, storage and network option. They even offer multiple public IP addresses for instances, which is not very common at all. At the backend, you get access to a private VLAN, KVM-over-IP, VPN access (PP2P or SSL). That is outstanding.
Like Amazon (S3, EBS) and Rackspace (CloudFiles) they also offer independent storage and distribution options (CloudLayer Storage, NAS, iSCSI even). And of course a CDN, too.
What’s very interesting with SoftLayer is the ability to mix actual physical hardware and CloudLayer instances on your private VLAN. That really offers opportunities, which you struggle to find elsewhere, especially as you can customise your network, IPs, Storage etc any time.
However, there’s one downside with SoftLayer: their pricing is certainly justified, but it’s the wrong place to go when you look at reducing costs. The smallest VM with 2GHz core, 1GB Ram, and 25GB storage together with 10MBit/s bandwidth will cost you about £50/month on an hourly schedule. Now, this isn’t very expensive, but extras like IPs, additional volumes etc weigh in significantly. It’s all the little extras which quickly let costs spiral. Most of those extras are charged on a monthly basis from what I gathered. On top of that your OS choice is limited (not as limited as Rackspace, but FreeBSD is not supported for example, and there don’t seem to be plans to change it.)
I’d like to add that starting instances does not happen instantly. In my tests it took more than 10 minutes. Too long for my taste, but probably good enough in most use cases.

ElasticHosts, with data centres in London, apparently use HVM virtualisation only. This allows enormous flexibility in OS choice (FreeBSD works, yay!), and gives you VNC access before the operating system even starts booting. You can run just about anything there. What I like is their subscriptions. For example you could reduce your costs by committing to an overall of 8GHz, 8GB, 80GB disk space, and then go and split it into 8 instances of 1GHz, 1GB, 10GB disk. That is quite neat, and their pricing is very attractive too.
That said, their interface is dreadful compared to others, and it times out frequently resulting in error messages. I wasn’t at all convinced by their network and disk performance either. The idea and concept are great, but the technical implementation can’t compete with the big guys, I think. Your mileage may vary.
What’s good there is that you can get VLANs, IPs, Firewall options etc, and you can download and upload your own OS images and CD-Roms via FTP.
I like their ideas, but it somehow doesn’t feel right, if that makes any sense, although they have been around for a couple of years.
But see for yourself. They offer a couple of days trial for free. (And don’t even ask you for credit card details.)

CloudSigma, with data centre in Zurich/Switzerland, are surprisingly similar to ElasticHosts in their terminology and also offer a free trial period. The subscription model works exactly the same way. They, too, use HVM and give you VNC access. And they also have FTP upload and download functionality to get hold of your VM images or put your own there. Apparently, CloudSigma haven’t been around very long (two years or so).
Their instances seem to perform better and disk I/O is faster than ElasticHosts’, too. Talking of disks, they even allow mounting the same volume to multiple VMs. However, that of course means you’d have to use an OS and filesystem, which is aware of clustering (OCFS2, GFS, CLVM). CloudSigma offer encryption for data at rest, which is very convenient.
Another big plus is the availability of SSD storage. I am yet to test FreeBSD with ZFS on their instances, utilising a SSD volume for ZFS’s Intent Log (ZIL). In theory that should rock, at least if SSDs are local and not served via network! However, during the trial they disallow SSD use, and to get started you have to topup at least £50, which would be the equivalent of two very small VMs for a whole month.

Neither ElasticHost nor CloudSigma offer services like S3 our CloudFiles. Also, you won’t find load balancers or DNS services there.

See, every single cloud is unique, and “The” Cloud simply doesn’t exist. One has to look very carefully at the offers and options. (This blog currently runs on FreeBSD in Amazon’s EC2, in case you wondered.)

I’m open for any other cloud provider suggestions to look into. If they match many of my requirements and have one or more data centres in Europe, I’m definitely interested! Please feel free to comment.

In case you haven’t heard: RedHat acquired Gluster back in October 2011, and that is very good news, because it will push development, broaden the user base, and ultimately make it more interesting for some of the bigger players out there, which will result in even better stability and performance altogether.

Frankly, there were times when GlusterFS was in a mess and the only people knowing how to configure and properly use it were the devs (or the freaks who hung out in IRC whole day, which I sometimes did too), because documentation was a nightmare. These things will now change (and have already changed as far as documentation is concerned). That’s brilliant progress.

If you don’t look at GlusterFS now, you’ll probably feel left out soon.

As CentOS 6.x has finally caught up with RHEL’s update cycle (as far as possible for a clone/copy/fork/you name it), I thought it might be good to have up-to-date images on EC2. But to my surprise I didn’t find many. Maybe it’s because you can get RHEL instances there, but I still prefer not to be dependent on licensing.

Anyhow, most AMIs I’ve found are outdated (6.0) or littered/bloated or heavily customised. I didn’t find a single AMI to my liking in either Europe or in the US regions. Plus, I wanted SELinux in enforced mode, which the vast majority of AMIs out there don’t offer and which is a bit of a nuisance to rectify later (due to long relabeling procedure and required reboot). Hence I created my own, which I made public.

The 32bit AMI works on t1.micro, m1.small and c1.medium out of the box. It’s wrapped into a 10GB EBS volume, 2GB of which are configured as swap space. The AMI is based on a groupinstall core (aka minimal installation). At startup it will gather your SSH key and chuck it into the authorized_keys2 file. The default user is ec2-user and root login is not allowed — going with Amazon’s best practices there.

If you prefer to log on as someone else, just type the username into the Userdata field when you create the instance via web console, or use the correct parameter (-d) with ec2-run-instances. I would for example just type carsten, and my SSH login will be carsten instead of ec2-user. I thought this would be a useful idea, if you run lots of instances and prefer to use your own login name. Old habits die hard as we know

The AMIs are publicly available as of now in both Ireland, California and Singapore:

• eu-west-1: ami-c5053cb1 (32 bit) and ami-c3053cb7 (64 bit)
• us-west-1: ami-2f9bc26a (32 bit) and ami-e792cba2 (64 bit)
• ap-southeast-1: ami-76b8fc24 (32 bit) and ami-72b8fc20 (64 bit)

Core features again:

• CentOS 6.2, no 3rd party repos, last update 15/Feb/2012 (32bit), 16/Feb (64bit)
• based on ‘groupinstall core’ (+ openssh_clients for scp)
• SELinux enabled and enforced
• swap preconfigured and part of the volume (/dev/xvde2)
• login user with sudo access either ec2-user or user specified at launch in the userdata field
• 32 bit version works on t1.micro, m1.small and c1.medium
• 64 bit version works on t1.micro and all other 64 bit instance types (not tested on the cc1, cc2 and cg1 monsters)
• ephemeral storage is made available, where the instance type permits (all but t1.micro); partitioning/formatting is your job

However: Use at your own risk! It’s a clean CentOS setup on EC2, no dirty trickery or other changes which require my support or make it a non-CentOS setup. Hence, use the normal support channels for questions related to CentOS (or use my contact form and have some cash available to spend )

Have fun and feel free to give feedback here or via contact form!

Now, I don’t want to spread rumours or create unnecessary panic. I don’t really doubt that the CentOS team will somehow manage to increase their pace a little bit, and their latest announcement regarding continuous releases (essentially “backported” security updates) goes into the right direction, if they can for once stick to their promised timelines.

Nonetheless, I need stability and consistency. That absolutely entails security updates, quite obviously. From my personal and professional experience, if it has to be Linux, RHEL-derivates are by far the best bet for enterprise environments. So, just in case, what else is in store for paranoid people like me who have committed to using RHEL and its forks/clones? (where FreeBSD sadly isn’t an option)

It’s not that I never heard of it before, but somehow I dismissed it as, well, scientific or academic: Scientific Linux. Probably I’m not the only one who was misguided by its name. The obvious questions are: Is it fully binary compatible with RHEL? What additions or modifications are included? Has anything important been removed? Who’s backing and supporting it? And: how up to date is it?

I had a close look at the website and repositories, and I was in for a very pleasant surprise actually: Scientific Linux is maintained by major scientific organisations, hence the name, and claims full binary compatibility with only very minor changes to the base installation of RHEL 6. The main goal of Scientific Linux (or “SL”) is to provide their users with an easy to customise RHEL-clone, which can be wrapped up into entirely new distributions (“Spins”). Also they provide a bit of entirely optional stuff, basically additions to the original. The important thing is: SL is a full clone, and it is entirely built from RHEL’s source RPMs (which can be found in SL’s repository of course, as the GPL requires).

How long has it been around, and how likely is it that it will last? Again a nice surprise: It’s actually older than CentOS, by about one year, and first appeared early 2004. And obviously it’s got the resources (and manpower) to keep it going. Their updates are released much faster than CentOS’s. For example for 6.0:  RHEL Nov/2010, SL Mar/2011, CentOS Jul/2011.  For 6.1: RHEL May/2011, SL Jul/2011, CentOS not yet available. Or for 5.7: RHEL Jul/2011, SL and CentOS both not released yet. However, SL has all the upstream updates available. The latest updates are from yesterday and include the issues in DHCP, Firefox et al, as announced by RedHat two days ago. None of these recent updates are in CentOS’s CR repository, despite the two-day old promise that said updates would be made available via CR within 24 hours. The latest CR updates are 5 days old. For me personally it doesn’t matter, because I’m not affected by the issues which were fixed since then; but others may be.

Don’t get me wrong. Five days is not a long time, especially as rolling everything out across the board will take a few days as well, given typical enterprise planning pace and decision making. However, it’s not really continuous and not in line with Karanbir’s own guesstimates.

Anyways. Time for some hands-on impressions. Or, wait, what do we expect to see in a binary-compatible RHEL-clone, which we haven’t seen in CentOS already? Exactly, despite some branding changes it is the same. The installed packages are identical as well, except the tiny differences mentioned above. In fact you could go install CentOS 6 and take the SL update repositories to update to more current packages (or SL 6.1). I’ve done that to prove my own theory. No surprises there. It’s not the most elegant way of updating CentOS, but certainly even less intrusive than using other third party repositories, and lesser trouble than building your own updates from RHEL’s source RPMs, believe me. Plus, SL’s updates originally come from RHEL.

Or… well… you could of course switch to SL altogether, given that you get essentially the same product, with shorter update delays. Taking the background, history, manpower and all that into account, the seemingly academic distribution actually looks a lot more enterprisy at the moment than CentOS does. I hope Karanbir Singh and his team can fill that gap very quickly. I’m not too keen on switching to SL, and I know that the CentOS team is very committed and doing a great job. But if we can’t get at least latest security updates for CentOS in a timely manner, it might become inevitable to switch.

That doesn’t mean that I will hastily switch the distributions of ~100 installations; nor should anybody else rush that decision. But come the time that we put 6.x in production, we might as well go for SL then. However, that’s certainly not going to happen this year any more, and requires careful side-by-side evaluation first, which is starting as I write this, and will last for months. How close we get to switching to SL seems to depend solely on CentOS; if they manage to get back to normal pace, this whole consideration may become obsolete. It’s good to know though that there are viable alternatives if need be.

So anyways, enough waffle here. Those of you who are interested, should follow this link: my step-by-step guide on how to set up a FreeBSD PV guest from scratch, including pygrub support. The tutorial will walk you through all the steps required from setting up an empty stub, over creating a HVM guest, to transforming that into a PV guest (or even hybrid, if you wish).

What is GlusterFS, and what is it not?

You can think of GlusterFS as a RAID device, which works across the boundaries of a single physical disk array. Take RAID-1 for example, which mirrors data between two identical disks. In GlusterFS’s jargon, you run two bricks in replicate mode, where a brick is defined as storage in general terms; it can be an array of disks (which could use RAID), a single disk, a partition, a directory. Anything that can be mounted into your filesystem hierarchy qualifies as a brick. The key feature of GlusterFS is now, to treat bricks on different physical machines as one volume, which can be accessed by any number of clients. It can be mounted either via Fuse/GlusterFS client, or even via NFS or CIFS/Samba.  You can use RAID-0 style striping for read speed, RAID-1 style mirroring for real-time replication, RAID-10 for both, or you can go beyond any of those and spread the stripes or mirrors across any number of bricks. 4-node replication? No problem at all. GlusterFS gives you truly enormous flexibility and performance when it comes to making large amounts of data available across multiple nodes.
Since version 3.2 (if I’m not mistaken), they have even added GeoReplication, which allows a Master/Slave setup, where the slave can be a local or remote site. Be it for backups or to have a standby version of your application in a different geographical location… it’s possible. Due to the fact that GeoReplication does not require locking or synchronous replication, the network speed to your remote site isn’t that important either. It copes well with it.

This sounds very different than for example a DRBD/GFS2 or DRBD/OCFS2 setup, doesn’t it? And indeed it is! GlusterFS, unlike DRBD, is not providing a block device. What it means is that it compares hashes of files, and if files on nodes differ (for example after a failure), it will copy entire files across, not only the changed blocks. In normal day-to-day operation that’s not a big problem, in particular as you get a lot of flexibility, which is unmatched by other solutions. Where it does make a difference is during recovery. More on that in the Caveats section.

A variety of different connectors

I mentioned earlier that you can use a couple of different ways to connect to your GlusterFS volumes. First, there’s their own GlusterFS client, which uses the kernel’s Fuse layer. This client is Gluster’s recommendation, if your workload requires a high amount of fast write operations. If your workload is more about reading small files quickly, they recommend NFS. (The NFS server is part of the glusterd daemon, which serves the volumes to the clients.) Samba/CIFS is probably mainly targeting Windows clients.

All these connectors have their advantages and disadvantages. You want to test that thoroughly for your particular workload. Also, in SELinux environments, you will require some tweaking of your policies, if you use the GlusterFS client, whereas NFS is a lot more straight forward (don’t forget that apache needs to be allowed to access NFS directly if that’s your intention; setsebool -P httpd_use_nfs=on is your friend). I know most people find it easier to switch off SELinux altogether, but for me personally that is never an option. I’d rather spend hours tweaking the SELinux policies, if necessary. Anyhow, the bottom line is that both NFS and CIFS make GlusterFS very attractive for platforms beyond Linux. FreeBSD for example, although I’m not sure if the native client has reached a production-ready state there yet; I shall give that a spin soon, and in the meantime NFS will do.

Performance

As a rule of thumb you can say that high-availabily, robustness, scalability etc always come with a downside: write performance. During write operations, all nodes need to be kept in sync, which means that the weakest “link” (or slowest disk for that matter) together with some locking and network/protocol overhead determines the actual write speed. That is normal. (Note: pure throughput must not be confused with the time it takes to actually be able to access a file on a different node than it was written to)

For that reason you can never expect a high availability file system to solve all your problems. There’s no such thing as “one size fits it all”. Your application need to be cluster/HA aware. In practice that means you will have to select carefully which type of information you store where. This is of course true for GlusterFS, too. However, when it comes to read performance, GlusterFS is actually very fast. Not as fast as a local block device, obviously, but personally I wasn’t able to tell the difference between native NFS and Gluster’s NFS implementation. The GlusterFS client (fuse/glusterfs, not NFS) however seems to be a little bit slower reading data, while being faster writing. It really depends on your workload. Bottom line is: GlusterFS is fast and flexible, which alone is a big plus over many other solutions. For maximum read performance you can of course use stripes (data scattered across multiple nodes), which the glusterfs client connects to simultaneously. It’s kind of obvious that in particular big files benefit from such a setup.

Caveats

If you intend to deploy GlusterFS, you better plan a serious amount of time for the first tests, integration into your setup, including benchmarks and failover. GlusterFS is powerful and not too difficult to get started with, but you’ll soon run into various rather specific questions, which aren’t documented well (or not at all). Quite frankly the online documentation is poor, or rudimentary. Obviously Gluster, a business, wants to sell their expertise, and there’s nothing wrong with it. So be prepared to browse mailing list archives or hang out in #gluster in irc.freenode.net or so.

GlusterFS has matured a lot over the last years, and you certainly don’t need to be worried about losing data (after all it’s filesystem based and you can copy anything out of the bricks’ directories directly, if you wish). However, some major issues and pitfalls still exist.

• If you reintroduce or replace a node, which was either faulty or offline for a while, the self-healing will transfer entire files back from up-2-date nodes onto the reintroduced one. This consumes a lot of network bandwidth, and even worse, CPU load (possibly due to the hash comparison). If a GlusterFS brick lives on a box together with other services, you will experience a significant performance hit.
• Large files are locked while being replicated. In practice that means that you really can’t use GlusterFS as a backend for VMs at the moment, unless recovery always happens in a controlled manner at times where you can afford to shut down running VMs for the entire duration of the healing. That somehow defeats the purpose of a high-availability storage cluster.
  However, a GlusterFS engineer has told me earlier today on irc.freenode.net that this issue will be tackled in GlusterFS 3.3, if not earlier. Only a question of months, I suppose.
• You absolutely must synchronise the system time of all bricks. If you’re not doing that already anyway, do it before deploying GlusterFS. (use NTP for your own sanity)
• Make sure that the bricks of one volume are of identical size and that you don’t by mistake fill the disk space by other means. I had a situation the other day where I wanted to replace a brick; what I didn’t realise first was that someone set a disk quota on the new brick. Consequently it stopped writing long before all data could be copied. However, GlusterFS did not warn me, nor did it report an error; it actually confirmed successful migration, although only 1/3 of the files were transferred!
  Clearly the lack of accessible disk space wasn’t GlusterFS’s fault, and is probably not a common scenario either, but it should spit out at least an error message. Imagine what would have happened if I had taken the other node offline after allegedly successful migration! Total mess.

Presumably none of these things would have happened, if I had taken their commercial offerings.  Those of you who prefer D.I.Y., better be prepared to spend a serious amount of time to fit it into your use-case and more importantly… monitor it closely!

Summary

GlusterFS has made a lot of positive progress over the last 2-3 years. It’s very easy to get started, especially on RHEL/CentOS, and it offers enormous flexibility and opportunities. The new CLI makes basic configuration much much easier than it used to be before. With a few simple commands you can create your volumes (on multiple servers, aka “peers”, simultaneously). You could say that it’s actually fun to use GlusterFS!

However, if you (like me) are looking at GlusterFS as a backend for Xen or VMware VMs in order to facilitate live-migration and resilience, you will probably need to wait for version 3.3, unless controlled recovery with planned downtime is an option for you. Might be worth keeping an eye on their Git repository (I certainly will). While using it to serve files for all sorts of things already, I’m really looking forward to using it as a backend for Xen soon!

Version 3.3 brings some other new promising features, too… Unified storage, object storage… I see memcached on the list of dependencies… looks promising. Beta 1 is out, by the way.

So where’s the but? Here it comes: …but if something unexpected happens, you are seriously screwed. Here are a few examples from the past couple of months.

Changing a NIC, which is also management interface, of a pool server — This was about the worst nightmare I’ve ever had. What you’d expect to do is: shutdown the machine, open it, replace the NIC, close it, switch it on again, wait for it to boot and start the VMs, done. What really happened is: I had to actually wipe and re-install the whole box, because there was apparently no documented, reverse-engineerable, or otherwise known way to just simply change the MAC address somewhere, because that is managed by the pool master. Now, as the NIC was broken, the master wasn’t able to communicate with the pool server any more (not even on the second NIC, because that was not the management interface). Attempts to change it failed. Not even the “xe” tool was functional any more, so I couldn’t really gather the UUIDs in order to search through configurations etc. The master refused to talk to the pool server, and the pool server with the broken (and afterwards replaced NIC) refused to let me change anything, because that should be done on the master. Catch 22.

I consulted the official support forum, but nobody knew an answer there either. I’m sure there is a way to change it easily. After all it’s a Linux box with a modified Xen, but still not an unaccessible blackbox. Hang on… actually it felt a bit like that. I would like to think that Citrix certainly knows an easy solution, but as I’m not paying thousands of Pounds for a product, which is almost entirely based on free software, they of course kept quiet. (The bloody toolstack, which complicated things, is their own development, by the way.)

End of that experience was that I had to remove the server from the pool (XenServer would then wipe the box, so you can’t re-join the pool later, either… awesome). After a clean setup and restoring all the VMs from previously created snapshots, the machine was finally able to join the pool. That was 6 hours after the NIC broke. Fortunately all VMs have an identical twin running on another machine, so it didn’t cause downtime (except a few minor hiccups while I was fiddling about with network settings). Otherwise all websites/applications would have been offline for 6 hours.

Without the XenServer toolstack, I could have resolved the issue within 10 minutes, which includes all of the steps mentioned earlier (what I would have expected).

I learned my lesson from it. As live-migration of VMs isn’t really necessary in most cases (my customers’ applications don’t benefit from it), it’s actually better to not form pools of your servers. Disconnected standalone servers are a lot easier to maintain and you don’t risk side-effects with pool members, because there aren’t any. The only real downside is that VLANs need to be configured individually on each server. Same applies to shared resources (NAS etc). But that’s fine.

Another almost unbelievable example is deleting snapshots. I create them all the time, because if something goes wrong, or someone breaks a VM setup, you want to be able to roll back to a previous version. Snapshots are one of the biggest advantages of virtualisation. A whole VM can be brought back to an older state within seconds. Or you can export it and reimport it elsewhere, clone another instance from it, work there, swing later. Anyways, if you use that feature often, it fills your disk (even the huge disks you get nowadays). So you regularly delete them and get your space back. Right? Nope, wrong. With XenServer you may or may not get your space back. When your monitoring tells you that you are running out of disk space, although you haven’t done anything but rotating snapshots in a while, you scratch your head in disbelief. Well, at least I did. Unfortunately, the official documentation confirms my observations. When I first read that reclaiming space causes downtime, I wasn’t sure if laughing or crying was the best course of action.

In a production environment, you can’t just go ahead and suspend VMs just to get space back. Even if you only reduce performance (without causing downtimes, as we’re running twins of everything), you need to make affected customers aware of it. And how do you explain that? “Sorry, Sir, I need to suspend your service, because I need to delete old snapshots.” They’ll think you’re taking the piss.

Again, this “feature” is brought to you by Citrix’s toolstack, not Xen. If I decide to delete an LVM-based snapshot of a running VM on Xen, I can do that any time. No need to suspend anything or to manually reclaim free space afterwards.

My favourite subject is VLANs. I don’t know how many hours I’ve wasted trying to find what I did wrong, just to figure out in the end that it was not my fault… Citrix apparently manipulated the bridge code and never really tested it. You have to actually install ebtables (iptables for bridges, if you will) to work around that issue. I observed exactly the same thing as the poster there, and many others did, too. Their forums are full of problems related to VLANs and NIC bonding. Problems get worse with two NICs. VLANs may work out-of-the-box on both, only one, or none of the NICs. Apparently it depends on the used NIC (well, I’m assuming here that nobody uses old NICs without VLAN support any more nowadays), which of the NIC is management interface, and a couple of other factors like weather, mood etc.

Once you know about the workaround mentioned earlier, you can solve it. But now, when you update your XenServer version, you can’t rely on Citrix. They might just remove the required kernel modules so that ebtables wouldn’t work any more. Sounds unlikely? Well, reality is that ebtables did work until XenServer version 5.5, but in 5.6 the kernel support was removed (see last post here). To fix it, you end up downloading the XenServer SDK (which includes all the open source bits they are using) and recompile the kernel yourself.

I won’t go deeper into this subjcet, but there are several issues with bonded NICs as well. And the management interface can never be on a tagged VLAN. All those are restrictions/problems solely related to Citrix’s stuff. Linux itself lets you create any combination of bonds and VLANs on as many interfaces as you want to. Unfortunately, you need to unlearn all about Linux network configuration, because if you try applying your knowledge, XenServer will overwrite your configuration as soon as you reboot (best case) or use its API or Windows client to manage NICs/VLANs.

I could go on and on and on. There are many other quirks like being unable to shutdown a VM when for some reason it can’t attach to a VNC console (but keeps trying, although you absolutely don’t need a console to shut it down); having a “force” option for many commands, which is useless, because it doesn’t force anything; being unable to remove stale shared storage; having to work around limitations which would for example disallow you to build a pool with an i7 920 and an i7 930 server; and quite a few more, which are of minor relevance in a production environment.

Don’t get me wrong. If you dig deep enough, you will find problems in any similarly complex software. And Citrix’s XenServer is not a bad product at all. Much of the functionality like live-migration isn’t available in VMware’s free version ESXi, and said free version doesn’t run on top of CentOS but on a custom Linux, which officially you can’t access via SSH (there are ways though, but you can’t expect any support at all). Also, XenServer’s GUI is self-explanatory and easy to use — and certainly one of the main reasons for using XenServer, because whoever is going to use it after you set it up for them, they won’t have many problems getting started.

However, if you don’t have lesser knowledgeable people using it later, and if you don’t mind going the extra mile, you probably get most flexibility and reliability if you set up Xen instead (the vanilla or “real” one, not XenServer). XenServer doesn’t really provide any additional functionality, which isn’t available in Xen. (Some people even say the opposite is true, and you only get full Xen functionality if you purchase XenServer’s extra licenses; I wouldn’t go that far.) It does add convenience with its GUI and toolstack though, which you’d otherwise have to implement yourself — snapshots, shared storage use, starting up any type of guest OS etc. Most of those things aren’t exactly rocket science; only a few are a bit more tricky. But you can script/automate them as you please and you don’t need to expect any bad surprises caused by 3rd parties.

For example, I disabled Xen’s bridging code (by commenting out a single line in their scripts) and do the whole network configuration with standard OS tools, keeping it independent and consistent for future updates. (More details here.) Snapshots are easy enough to do with LVM, too. Live-migration I haven’t tested yet, but it doesn’t look too difficult to do either. (We don’t really need that feature here anyway)

What I’ve struggled with was getting different OS running, namely FreeBSD. But now that I have sorted that out, I can easily clone and fork more FreeBSD VMs on the vanilla Xen machines. Hence, Citrix XenServer isn’t providing any benefits there either.

As you can see (and as the title suggests), I’m considerably fed up with XenServer’s quirks; some of them are too huge to accept them in production environments. Consequently, we’re going to “migrate” back to Xen, where we can. (Admittedly, in some environments we won’t be able to do that for another year or so.)

Once you’ve worked out how XenServer stores VM backups (yep, they did their own thing there too, and the format is really stupid), it’s not too difficult to convert them. I’ve done that for both CentOS and FreeBSD XenServer images. They run smoothly on vanilla Xen after converting them back.

Once again the “keep it simple” motto wins. Additional toolstacks and bloat cause more problems than necessary, and the manufacturer turns out to be the only one benefitting from it — as often is the case. So long, XenServer — Hello Xen!

(Update: Only three hours after I published this, one of our XenServers started refusing to create new VMs from templates…)

(Update 2: It’s cursed. Yesterday I was all of a sudden unable to attach any block devices, hence I was unable to start new VMs, reboot existing ones, or increase storage. I’m not the only one, who faces that problem and does not get any help from the experts at Citrix.)

(Update 3, Aug 25th: Done. Last weekend we’ve transformed the last remaining XenServers to vanilla Xen. Thanks to the twin-design, this went through without any downtimes whatsoever; was a major piece of work though, but certainly worth it. Chapter closed. )

Admittedly the learning curve for both is quiet steep, and everyone’s well-advised to spend some serious time evaluating them. When I started looking into EC2, I only had a rough idea of all the services they offer. I was quite overwhelmed, how many related services EC2 (or more precisely AWS) entails:

• EC2, the cloud, which runs your instances (also known as virtual machines, Xen based) in one out of four regions (US East/West, EU, APAC) and one out of two-four availability zones in each region
• Elastic Load Balancing (ELB), giving you the opportunity to spread load across instances, obviously
• Elastic IPs, allowing you to assign (and re-assign) static IPs to instances of your choice
• Simple Storage (S3), which guarantees replication of your stored data in three different locations, enabling it to survive an outage of two entire data centres (or one data centre, if you opt-in for the “reduced redundancy” option, which is a little bit cheaper — you can choose that for every file stored individually)
• EBS (Elastic Block Storage), enabling you to create RAID-backed volumes of any size and attach them to any of your EC2 instances; on top of that you can create snapshots (which are internally stored on S3) within seconds
• RDS (Relational Database Service), basically a MySQL offering, in either single, single/hot-standby, master/slave, or master/multi-slave setups, with nodes spread across different availability zones
• CloudWatch, which entails monitoring facilities for most of the services
• CloudFront, a multi-region CDN-like service
• SimpleDB, Map/Reduce
• Route 53 DNS services (beta)
• DevPay, Flexible Payments

All these services have one thing in common: They can be managed entirely via different APIs and command line tools. There’s nothing which you can’t automate, if you spend some time and effort to actually understand how it all fits together! It’s certainly very overwhelming in the beginning, and Amazon clearly doesn’t target customers who might want to fire up one or two instances and that’s it. It’s way too complex for that. And it requires an entirely different approach, for example an instance and all its data is lost when you terminate it. And all resources are very dynamic — for most simple use-cases too dynamic (you don’t know which IP or hostname your instance will have; most provided OS images won’t suit your needs, so you’ll need to build your own). But if you are interested in creating environments for your applications, which come with both high availability and scalability, then EC2 is definitely worth a shot. Amazon gives you the bullet-proof and battle-proven infrastructure and tools — you need to decide and find a way how to use them for your requirements.

Amazon offer the AWS console for very basic management of your resources. Very basic. You’ll soon find out that it can’t offer things, which you really will need:

• creating a snapshot, which you can use to boot another instance from (or as a backup to start the same instance again, when it fails)
• setting triggers for the CloudWatch monitoring (or alarms as they call it in their API)
• bundling your instance (or parts of it) and backing up on S3
• moving instances between availability zones
• configuring the RDS MySQL server
• and many more things

All of these things can be done via API (in Java, PHP, and other languages, or via command line tools, which can all be downloaded from Amazon). Some of them are trivial, most are not. Flexibility takes its toll. Consequently, you should be prepared to spend some time tailoring your own toolset. There are some 3rd party offers out there (notably the best on is s3cmd, which allows rsync-style file transfers between instance and S3 buckets). They may or may not suit your needs.

I’ve spent the last two weeks creating my own toolset. With very simple commands I can now build fully bootable AMI images for different Linux setups both in 32bit or 64bit (EC2 instance types differ in terms of architecture!), create bootable snapshots from running instances, detect instance failure and restart from the most recent snapshot (including re-assigning the elastic IP), set tags and other information/attributes on all sorts of resource types, create volumes (empty or from snapshot) and attach them to instances, hook instances into a load balancer, read all relevant CloudWatch metrics and feed them into RRD graphs, clone instances on-the-fly, launch any number of clones, manage security groups and keypairs etc.  Basically everything the AWS console can plus a few necessary features on top of that — with a single shell command and no more than 2-3 parameters each. I’m not exactly a developer and started doing this merely as a proof of concept (but then went further than originally intended to). If I can do that, some of you bright-minded developers can do a lot better for sure

This was AWS management covered. But how about managing the actual instances (their OS internals)? What if, for example, you want to deploy a web application on four identical, load-balanced nodes?  Should I create a dedicated image for that (not too difficult with my toolset)? Or would it be better to have a look into Puppet at last? I went for the latter. I’ve got customers on my own clusters outside of EC2 (mostly based on Citrix XenServer), and that environment is growing continuously. It’s about time that I simplified management there as well.

Consequently, I decided to take my EC2 proof of concept another step further. After getting acquainted to Puppet, I’ve deployed it on a playground-style bunch of EC2 instances and told it to install/configure various things. The language structure really gave me hard times in the beginning, but once you get used to it, you can almost write it down as you think.

The next thing I wanted to achieve was that puppet connects to the puppet master as soon as the instance is started. There were some obstacles in the way, though: AWS assigns hostnames dynamically, but your puppet master would need to know that hostname in order to sign the certificate used for communication between both. Catch 22 situation. Resolved by writing a tiny web service which allows the instance to figure out and set the hostname I assigned (and dynamically added to a DNS server as well) rather than using Amazon’s one. This happens during startup just after the network interface comes up, so that all running services use the correct hostname. Puppet then takes over at the end of the first startup of the instance and installs/configures as told by the puppet master. This way you can fire up a whole cluster, hook it into the load balancer, and are ready to go live in just under three minutes. Fully automated. And the monitoring mentioned earlier would pick up metrics via CloudWatch instantly.

I’ve heard it all in theory before. However, I wanted to see my own working proof of concept for a few things (and some others, which are still in progress). I’m pretty amazed actually, how much flexibility and reliability (often a contradiction in terms) AWS offers. You just have to embrace a slightly different model of implementing things (you’ll like the term “ephemeral”, which Amazon have chosen for a reason!).

So now the next question would be: How much does it cost. Is it really saving costs as many people state? Frankly, I don’t know yet. It may do. Surely, it reduces upfront costs, as Amazon won’t charge any setup or recurring fees, unless you opt-in for their “Reserved Instance” schemes, which are actually significantly cheaper in the long run. For example, a Micro instance (640 MB Ram, 1.7GHz Xeon; the smallest instance type) would usually be charged at US$ 0.025 per hour, which equals to US$ 219 per year if running full-time. If you commit to a year, paying US$ 54 one-off, your hourly rate is reduced to US$ 0.01, which together equals to yearly costs of US$ 117.60 or less than US$ 10 per month! That’s a smashing 50% discount almost. Higher discounts possible, if you can commit to 3 years.

However, the pricing is somewhat difficult to decipher and costs impossible to predict. I don’t actually know yet, how many IOPs (I/O operations) my EBS volumes and snapshots will generate. I can’t exactly tell how much S3 storage I will use. Also, I don’t know exactly what to expect on the inter-availability-zone traffic scale. Or the RDS (MySQL) read/write operations. Surely, previous monitoring gives me very rough estimates, but not good enough to make an educated guess as to what costs to expect on EC2. I will have to keep an eye on that over the next weeks and months, and also find some tools to get all these figures from the usage reports (CSV or XML files, downloadable from Amazon). At least you can see how your usage translates into actual costs for the current billing period, updated every few hours. So the costs wouldn’t hit you as a big surprise

On the plus side, you never pay for any over-capacity, which you would need to account for, if you built everything in-house. When you build infrastructure like that on your own, there are different things, which scale more or less dynamically (if you’ve got 10 servers already, buying two more doesn’t do a harm). But you’ve also got devices where upfront costs are enormous, because you buy them from the point of view, of what you might need in the foreseeable future, not what you do need at this very moment. Storage devices are a good example. A chassis from NetApp with only a few drives cost you an arm and a leg; then you can scale it a bit for a reasonable price; and then you’ll need another one sooner or later. But you always end up paying for more than you actually use at any given point. Same for networking devices.

Clouds like EC2 take those massive entry-costs from you (and your customers), which saves painful budget discussions. They’ve got a brilliant, scalable infrastructure, and one would be bold to assume that you could build anything better at a reasonable price (also take availability in distinct and independent data centres into account!). Now that they’ve got their PCI DSS certification, one of the biggest remaining concerns (what about data security in a proprietary, shared environment?) for many customers is gone, too.

I’ll go through the other proofs of concept on my list, see how usage translates into actual costs over time, and may then be able to add some very interesting offers to my company’s portfolio. Stay tuned

Surely, the cloud is not the solution to all problems (although it’s commonly propagated as that), but with decent automation and tools it can improve or at least add value to a variety of services.

That said, we must not forget Ubuntu’s focus, which I think (and I will expand on it later) is still valid: desktop and laptop computers.

Ubuntu aims to bring the latest drivers and technologies to desktops (I will use desktop as a term for desktop computers, laptops, and netbooks here). It has to, because otherwise it won’t be able to compete with proprietary operating systems (read: Windows and Mac OSX). To achieve that, it has to put the GPL/non-GPL debate (which is a big issue for Debian et al) aside. There’s a bunch of repositories of not exactly free (or not even open source) software, which is essential to get certain hardware (e.g. graphics cards) and software (e.g. media codecs) working: Restricted, Multiverse, Universe, Medibuntu, etc. Although they are not officially supported, all of them except Medibuntu are included in /etc/apt/sources.list and active, plus they reside on *.ubuntu.com servers. So it’s a bit difficult to not consider them part of Ubuntu, or at least part of the Ubuntu-Conquers-The-Desktop success, which makes the discussion of “who’s responsible for what?” a bit more difficult to answer. But it’s a crucial question in an enterprise setup. This is just one example why I think that Ubuntu is not targeting enterprise server environments, and you can’t be the best choice for something, which you are not focused at. More further down…

I’ve just installed Ubuntu Server 10.4 LTS in a virtual machine here to verify whether my past experience still holds true. I went for the Install Ubuntu Server option, and only used defaults (except that I added OpenSSH). So except stated otherwise, I will refer to this version, which is the latest release for servers and allegedly targets enterprises.

The intention of this article is not to compare Linux distributions with each other or give any recommendations as to which Linux distribution is the best one to go for in an enterprise environment. It’s not my intention to badmouth Ubuntu or say that it’s not suitable for servers at all, either. I’m merely explaining why I’m not a big fan of Ubuntu, as I’ve been asked that question a couple of times recently. Okay, maybe I’ve provoked that question a little.  It’s no secret though that my favourite Linux distribution for servers is CentOS, if it has to be Linux, or FreeBSD, if the scenario permits and the operating system decision is a matter of what we want to achieve rather than what we want to use. But again, that’s a separate discussion and beyond the scope of this article. Also, there’s no “one size fits it all”. I have noticed that many people stick to the things they know or like best in many situations, where another operating system or Linux distribution might have been more suitable for a certain job. Although I can’t scientifically prove it, this seemingly applies to many people who use or did use Ubuntu on desktops. Maybe we should use the “fanboi” term not only for Apple’s repeat customers, but also for Ubuntu users

Okay, back to the original question, why I don’t like Ubuntu on servers…

Let me first define what my expectations are:

1. There is no such thing as one single server. Servers come in pairs at the very least. I build environments which are as fail-safe as possible (and affordable), load-balanced, robust.
2. Implementing the very latest developments and technologies usually does more harm than good, because they can’t have been tested by as many people as older features. I prefer well-tested, solid operating systems. If I really need a more up-to-date version of, say, PHP, then I build a package for that. I don’t need the entire distribution to include the latest features just because I need only one package to be a bit more up-to-date! (NB: I am talking about feature updates here, not security patches!)
3. I expect the operating system to provide reasonable security standards and default settings and leave the rest to me.
4. I prefer using established standard tools and best practices over “Mate, we’ve quickly put together a new tool for you”
5. I decide what is installed and what isn’t. I don’t need the OS to tell me what it thinks is good for me.
6. Most of all I expect a proper release cycle and thorough testing before labelling something as a final release. (Oh, I did mention that before, didn’t I? )
7. I don’t like operating systems or derivates, which are entirely built on top of an existing one. Additional layers cause additional dependencies, often inherit errors, and make it more difficult to track down where an error comes from, and who has introduced it.

Let me start with 7., because I hear you saying “But…”.  No, CentOS is not built on top of RedHat! It’s a 100% clone minus proprietary stuff, logos, and license/support costs. Ubuntu however is derived from Debian and has added loads of stuff, which includes many things that Debian refuses to include (e.g. proprietary drivers and non-GPL code in general), while incorporating lots of Debian packages. Remember this severe OpenSSL bug exactly two years ago? What happened was that Debian broke the random number generator (making keys predictable) in their OpenSSL package.  The only distributions affected were Debian and all derivates including Ubuntu, but not RedHat or clones/derivates thereof. I don’t blame Ubuntu for inheriting broken code, because nobody can possibly read and understand the source code of everything. However, that was when I lost trust in Debian (the code change was an utterly stupid attempt to get rid off compiler warnings without understanding what the code does), and as it is the foundation of Ubuntu, I can’t trust it either. You may call it nitpicking, but making changes (and introducing bugs) to crucial security related features, which would definitely not have gotten the upstream’s approval, if they had pushed it upstream, is pretty bad stuff. All SSH keys had to be re-generated and SSL certificates replaced. Not  a big deal for only a bunch of servers, but a massive amount of work for an enterprise.

Let me continue traversing the list above. Number 6: Releases. First of all, before installing a new update, I would like to be able to assess what changes will occur to my systems. That’s what release notes are for. However, if you are on the Ubuntu Server home page and click on Resources and then a bit further down on Release Notes, you in fact end up only with known issues for both Ubuntu Desktop and Server. It takes quite a while to find the actual key specs at least, hidden somewhere in the wiki. But I wanted to elaborate on release cycles…

From a server Linux distribution I would expect that it has been presented to a huge group of users prior to its final release. Ideally it goes through various beta or pre-release cycles, giving the users time to test (some things need time to test them properly) and developers time to fix issues. Ubuntu however sets deadlines: every April and October of each year, there’s has to be a major release. In other words: In a half-year cycle new features have to be selected, introduced, and tested. It doesn’t seem to be top priority to have rock-solid releases. Let me quote an Ubuntu developer:  ”And remember that, since this is a long-term supported (LTS) release, there are ample opportunities for further bugfixes after the final release by way of the SRU process[2]. Point releases for Ubuntu and Kubuntu LTS will be made at roughly six-month intervals, with the first expected in July 2010 to address any critical issues not identified or fixed in time for the 10.04 LTS release.”

I’m sure he didn’t mean it, but it sounds like: “Hurry up. Doesn’t matter if we can’t fix things on time, as we’ll come up with a bugfix release in July anyway.”  Beta 2, release candidate, and final release were published within only three weeks, by the way. Ubuntu, Ubuntu Server, and Kubuntu at the same time. It does raise questions, doesn’t it?

If you look at FreeBSD, just to compare two entirely different release policies, you’ll find that they first work out what issues need to be addressed and which features may be introduced. Then they come up with a very rough schedule. And then, after they have frozen the code, they go through many stages for major releases: BETA 1-4, Documentation updates, Release Candidate 1-3, Release. From the code freeze (except for bug fixes) to the actual release of 8.0 in November 2009, it took them 4.5 months. And, as usual, the result is a rock-solid operating system. The minor release 8.1 is planned for July this year (but not yet announced for a good reason). I expect it to be available in September or so. In my opinion, it’s much more important to get the issues solved rather than sticking to a fixed deadline.

Number 5: I’m the boss! As I said earlier, I just installed 10.04 LTS Server here. Although I did not select any packages except OpenSSH, I ended up with an installation eating 818 MB on my disk. Hello? It turned out that a whole pile of useless stuff is installed by default: Wireless support, PPP (yeah, good old dial-up!) support, file system support for NTFS and FAT32, tools to compile C/C++ etc. Seriously, that’s not funny. So the first thing I will have to do is remove all the litter (or scroll through lists of useless crap at install time and deselect there).

Also, I can’t remember that I have been asked whether or not I wanted AppArmor installed. I don’t! SELinux has been in the mainline Linux kernel since 2003. I don’t want that to be removed and replaced with another solution. At least I would like to have a choice. (However, I do embrace that Ubuntu comes with AppArmor now, which is still better than Debian’s and Ubuntu’s ignorance towards SELinux or any other security implementations over the last couple of years.)  Although iptables is available, per default it’s disabled. But instead they have the cool “ufw” tool, a front-end to the netfilter firewall, as they call it. What it does is using OpenBSD’s pf syntax to create rules for iptables. I guess I should like that, because pf’s syntax makes a lot more sense than iptable’s. Unfortunately, I don’t like any “front-ends” messing with my settings. On Linux, I expect to use iptables as the common standard. OpenBSD’s pf (packet filter) can be found on OpenBSD, NetBSD and FreeBSD. So again, I have to remove unnecessary stuff.

Furthermore, in the enterprise section, I would expect thoroughly tested support for DRBD, GFS2, heartbeat, haproxy et al in order to build solid clusters. However, GFS2 is marked experimental in Ubuntu 10.04. So it has not been tested properly in Ubuntu, which is a shame, because it has been on RedHat Enterprise Linux, where it comes from. And as GFS2 is one of the very few cluster-aware filesystems on Linux, I kind of would expect that to be thoroughly tested (GFS and GFS2 have been out there for years). Or why did Ubuntu Server claim to be an enterprise Linux again? Oh right, must have something to do with the Gentlemen’s agreement between Amazon and Ubuntu to exclusively ease access to Amazon EC2, a proprietary “cloud” (don’t get me started on this term). So what Ubuntu users get is an increasingly strong mix of GPL stuff with proprietary extensions.

Again, I’m not saying that Ubuntu is bad. And I really do appreciate Mark’s effort to create a very good desktop Linux, which keeps up with recent technology development and hardware support. For the server, on the other hand, I am a bit more conservative. I don’t need half-baked support for quite literally everything there. Nor do I need the very latest libraries and features. What I do need is robustness. I prefer a minimal base installation (which includes standard tools and security measurements) and to take it from there. And I prefer things which have been really thoroughly tested. Experimental is a word I don’t really want to read there. Bottom line is that Ubuntu Server feels a bit like an experimental server Linux for beginners.

That, my friends, is why I’m not fond of Ubuntu. Admittedly, I got a bit carried away here. And I do know that many of you (especially the Ubuntu “fanboi” folks ) will disagree. At the end of the day, every systems administrator has got their own preferences. Each to their own. No Ubuntu for me (unless I’m being forced to).

Now bring on the stones you want to throw at me…

Let’s say you are running FreeBSD on a server, and you need to do a major upgrade (that is from 6.x to 7.x). This process can take ages, if your machine is not running the latest hardware, and/or you have a lot of 3rd party software installed (ports). I’m not talking about an impatient person’s definition of ages, or about the one of a customer, who claims hundreds of quid financial loss in 20 minutes downtime on Sunday morning 1:30 am.  I’m talking about ages as in many hours.

Of course, a FreeBSD upgrade doesn’t require to be offline while it’s proceeding. But you will need to reboot. And as a rule of thumb, one can assume that dependencies in the ports will break. Usually only one or two of them, but it requires manual work, and can cause an unpredictable partial downtime, which is longer than it takes to reboot the machine.

So how can virtualisation help here? In a nutshell, it allows you to do the whole upgrade on another virtual machine. You can take a snapshot of the production machine, start it as a new VM, and do your work there, while the original VM stays online.

This also reduces stress enormously, because if you break something during the upgrade, there’s no time pressure to fix it. You can spend as much time as it takes to finish your work properly. Cool, isn’t it?

And when you’ve finished your work, you can inform your customer about an upcoming 1 or 2 minutes downtime for a major system upgrade (which you have already finished).

All you need to do when the time has come, is to sync files which changed during run-time (for example mail folders), change the network settings in order to make your upgraded snapshot take over, and then you can safely decommission the old VM. It really is as easy as that.