Now, I don’t want to spread rumours or create unnecessary panic. I don’t really doubt that the CentOS team will somehow manage to increase their pace a little bit, and their latest announcement regarding continuous releases (essentially “backported” security updates) goes into the right direction, if they can for once stick to their promised timelines.

Nonetheless, I need stability and consistency. That absolutely entails security updates, quite obviously. From my personal and professional experience, if it has to be Linux, RHEL-derivates are by far the best bet for enterprise environments. So, just in case, what else is in store for paranoid people like me who have committed to using RHEL and its forks/clones? (where FreeBSD sadly isn’t an option)

It’s not that I never heard of it before, but somehow I dismissed it as, well, scientific or academic: Scientific Linux. Probably I’m not the only one who was misguided by its name. The obvious questions are: Is it fully binary compatible with RHEL? What additions or modifications are included? Has anything important been removed? Who’s backing and supporting it? And: how up to date is it?

I had a close look at the website and repositories, and I was in for a very pleasant surprise actually: Scientific Linux is maintained by major scientific organisations, hence the name, and claims full binary compatibility with only very minor changes to the base installation of RHEL 6. The main goal of Scientific Linux (or “SL”) is to provide their users with an easy to customise RHEL-clone, which can be wrapped up into entirely new distributions (“Spins”). Also they provide a bit of entirely optional stuff, basically additions to the original. The important thing is: SL is a full clone, and it is entirely built from RHEL’s source RPMs (which can be found in SL’s repository of course, as the GPL requires).

How long has it been around, and how likely is it that it will last? Again a nice surprise: It’s actually older than CentOS, by about one year, and first appeared early 2004. And obviously it’s got the resources (and manpower) to keep it going. Their updates are released much faster than CentOS’s. For example for 6.0:  RHEL Nov/2010, SL Mar/2011, CentOS Jul/2011.  For 6.1: RHEL May/2011, SL Jul/2011, CentOS not yet available. Or for 5.7: RHEL Jul/2011, SL and CentOS both not released yet. However, SL has all the upstream updates available. The latest updates are from yesterday and include the issues in DHCP, Firefox et al, as announced by RedHat two days ago. None of these recent updates are in CentOS’s CR repository, despite the two-day old promise that said updates would be made available via CR within 24 hours. The latest CR updates are 5 days old. For me personally it doesn’t matter, because I’m not affected by the issues which were fixed since then; but others may be.

Don’t get me wrong. Five days is not a long time, especially as rolling everything out across the board will take a few days as well, given typical enterprise planning pace and decision making. However, it’s not really continuous and not in line with Karanbir’s own guesstimates.

Anyways. Time for some hands-on impressions. Or, wait, what do we expect to see in a binary-compatible RHEL-clone, which we haven’t seen in CentOS already? Exactly, despite some branding changes it is the same. The installed packages are identical as well, except the tiny differences mentioned above. In fact you could go install CentOS 6 and take the SL update repositories to update to more current packages (or SL 6.1). I’ve done that to prove my own theory. No surprises there. It’s not the most elegant way of updating CentOS, but certainly even less intrusive than using other third party repositories, and lesser trouble than building your own updates from RHEL’s source RPMs, believe me. Plus, SL’s updates originally come from RHEL.

Or… well… you could of course switch to SL altogether, given that you get essentially the same product, with shorter update delays. Taking the background, history, manpower and all that into account, the seemingly academic distribution actually looks a lot more enterprisy at the moment than CentOS does. I hope Karanbir Singh and his team can fill that gap very quickly. I’m not too keen on switching to SL, and I know that the CentOS team is very committed and doing a great job. But if we can’t get at least latest security updates for CentOS in a timely manner, it might become inevitable to switch.

That doesn’t mean that I will hastily switch the distributions of ~100 installations; nor should anybody else rush that decision. But come the time that we put 6.x in production, we might as well go for SL then. However, that’s certainly not going to happen this year any more, and requires careful side-by-side evaluation first, which is starting as I write this, and will last for months. How close we get to switching to SL seems to depend solely on CentOS; if they manage to get back to normal pace, this whole consideration may become obsolete. It’s good to know though that there are viable alternatives if need be.

So anyways, enough waffle here. Those of you who are interested, should follow this link: my step-by-step guide on how to set up a FreeBSD PV guest from scratch, including pygrub support. The tutorial will walk you through all the steps required from setting up an empty stub, over creating a HVM guest, to transforming that into a PV guest (or even hybrid, if you wish).

What is GlusterFS, and what is it not?

You can think of GlusterFS as a RAID device, which works across the boundaries of a single physical disk array. Take RAID-1 for example, which mirrors data between two identical disks. In GlusterFS’s jargon, you run two bricks in replicate mode, where a brick is defined as storage in general terms; it can be an array of disks (which could use RAID), a single disk, a partition, a directory. Anything that can be mounted into your filesystem hierarchy qualifies as a brick. The key feature of GlusterFS is now, to treat bricks on different physical machines as one volume, which can be accessed by any number of clients. It can be mounted either via Fuse/GlusterFS client, or even via NFS or CIFS/Samba.  You can use RAID-0 style striping for read speed, RAID-1 style mirroring for real-time replication, RAID-10 for both, or you can go beyond any of those and spread the stripes or mirrors across any number of bricks. 4-node replication? No problem at all. GlusterFS gives you truly enormous flexibility and performance when it comes to making large amounts of data available across multiple nodes.
Since version 3.2 (if I’m not mistaken), they have even added GeoReplication, which allows a Master/Slave setup, where the slave can be a local or remote site. Be it for backups or to have a standby version of your application in a different geographical location… it’s possible. Due to the fact that GeoReplication does not require locking or synchronous replication, the network speed to your remote site isn’t that important either. It copes well with it.

This sounds very different than for example a DRBD/GFS2 or DRBD/OCFS2 setup, doesn’t it? And indeed it is! GlusterFS, unlike DRBD, is not providing a block device. What it means is that it compares hashes of files, and if files on nodes differ (for example after a failure), it will copy entire files across, not only the changed blocks. In normal day-to-day operation that’s not a big problem, in particular as you get a lot of flexibility, which is unmatched by other solutions. Where it does make a difference is during recovery. More on that in the Caveats section.

A variety of different connectors

I mentioned earlier that you can use a couple of different ways to connect to your GlusterFS volumes. First, there’s their own GlusterFS client, which uses the kernel’s Fuse layer. This client is Gluster’s recommendation, if your workload requires a high amount of fast write operations. If your workload is more about reading small files quickly, they recommend NFS. (The NFS server is part of the glusterd daemon, which serves the volumes to the clients.) Samba/CIFS is probably mainly targeting Windows clients.

All these connectors have their advantages and disadvantages. You want to test that thoroughly for your particular workload. Also, in SELinux environments, you will require some tweaking of your policies, if you use the GlusterFS client, whereas NFS is a lot more straight forward (don’t forget that apache needs to be allowed to access NFS directly if that’s your intention; setsebool -P httpd_use_nfs=on is your friend). I know most people find it easier to switch off SELinux altogether, but for me personally that is never an option. I’d rather spend hours tweaking the SELinux policies, if necessary. Anyhow, the bottom line is that both NFS and CIFS make GlusterFS very attractive for platforms beyond Linux. FreeBSD for example, although I’m not sure if the native client has reached a production-ready state there yet; I shall give that a spin soon, and in the meantime NFS will do.

Performance

As a rule of thumb you can say that high-availabily, robustness, scalability etc always come with a downside: write performance. During write operations, all nodes need to be kept in sync, which means that the weakest “link” (or slowest disk for that matter) together with some locking and network/protocol overhead determines the actual write speed. That is normal. (Note: pure throughput must not be confused with the time it takes to actually be able to access a file on a different node than it was written to)

For that reason you can never expect a high availability file system to solve all your problems. There’s no such thing as “one size fits it all”. Your application need to be cluster/HA aware. In practice that means you will have to select carefully which type of information you store where. This is of course true for GlusterFS, too. However, when it comes to read performance, GlusterFS is actually very fast. Not as fast as a local block device, obviously, but personally I wasn’t able to tell the difference between native NFS and Gluster’s NFS implementation. The GlusterFS client (fuse/glusterfs, not NFS) however seems to be a little bit slower reading data, while being faster writing. It really depends on your workload. Bottom line is: GlusterFS is fast and flexible, which alone is a big plus over many other solutions. For maximum read performance you can of course use stripes (data scattered across multiple nodes), which the glusterfs client connects to simultaneously. It’s kind of obvious that in particular big files benefit from such a setup.

Caveats

If you intend to deploy GlusterFS, you better plan a serious amount of time for the first tests, integration into your setup, including benchmarks and failover. GlusterFS is powerful and not too difficult to get started with, but you’ll soon run into various rather specific questions, which aren’t documented well (or not at all). Quite frankly the online documentation is poor, or rudimentary. Obviously Gluster, a business, wants to sell their expertise, and there’s nothing wrong with it. So be prepared to browse mailing list archives or hang out in #gluster in irc.freenode.net or so.

GlusterFS has matured a lot over the last years, and you certainly don’t need to be worried about losing data (after all it’s filesystem based and you can copy anything out of the bricks’ directories directly, if you wish). However, some major issues and pitfalls still exist.

• If you reintroduce or replace a node, which was either faulty or offline for a while, the self-healing will transfer entire files back from up-2-date nodes onto the reintroduced one. This consumes a lot of network bandwidth, and even worse, CPU load (possibly due to the hash comparison). If a GlusterFS brick lives on a box together with other services, you will experience a significant performance hit.
• Large files are locked while being replicated. In practice that means that you really can’t use GlusterFS as a backend for VMs at the moment, unless recovery always happens in a controlled manner at times where you can afford to shut down running VMs for the entire duration of the healing. That somehow defeats the purpose of a high-availability storage cluster.
  However, a GlusterFS engineer has told me earlier today on irc.freenode.net that this issue will be tackled in GlusterFS 3.3, if not earlier. Only a question of months, I suppose.
• You absolutely must synchronise the system time of all bricks. If you’re not doing that already anyway, do it before deploying GlusterFS. (use NTP for your own sanity)
• Make sure that the bricks of one volume are of identical size and that you don’t by mistake fill the disk space by other means. I had a situation the other day where I wanted to replace a brick; what I didn’t realise first was that someone set a disk quota on the new brick. Consequently it stopped writing long before all data could be copied. However, GlusterFS did not warn me, nor did it report an error; it actually confirmed successful migration, although only 1/3 of the files were transferred!
  Clearly the lack of accessible disk space wasn’t GlusterFS’s fault, and is probably not a common scenario either, but it should spit out at least an error message. Imagine what would have happened if I had taken the other node offline after allegedly successful migration! Total mess.

Presumably none of these things would have happened, if I had taken their commercial offerings.  Those of you who prefer D.I.Y., better be prepared to spend a serious amount of time to fit it into your use-case and more importantly… monitor it closely!

Summary

GlusterFS has made a lot of positive progress over the last 2-3 years. It’s very easy to get started, especially on RHEL/CentOS, and it offers enormous flexibility and opportunities. The new CLI makes basic configuration much much easier than it used to be before. With a few simple commands you can create your volumes (on multiple servers, aka “peers”, simultaneously). You could say that it’s actually fun to use GlusterFS!

However, if you (like me) are looking at GlusterFS as a backend for Xen or VMware VMs in order to facilitate live-migration and resilience, you will probably need to wait for version 3.3, unless controlled recovery with planned downtime is an option for you. Might be worth keeping an eye on their Git repository (I certainly will). While using it to serve files for all sorts of things already, I’m really looking forward to using it as a backend for Xen soon!

Version 3.3 brings some other new promising features, too… Unified storage, object storage… I see memcached on the list of dependencies… looks promising. Beta 1 is out, by the way.

So where’s the but? Here it comes: …but if something unexpected happens, you are seriously screwed. Here are a few examples from the past couple of months.

Changing a NIC, which is also management interface, of a pool server — This was about the worst nightmare I’ve ever had. What you’d expect to do is: shutdown the machine, open it, replace the NIC, close it, switch it on again, wait for it to boot and start the VMs, done. What really happened is: I had to actually wipe and re-install the whole box, because there was apparently no documented, reverse-engineerable, or otherwise known way to just simply change the MAC address somewhere, because that is managed by the pool master. Now, as the NIC was broken, the master wasn’t able to communicate with the pool server any more (not even on the second NIC, because that was not the management interface). Attempts to change it failed. Not even the “xe” tool was functional any more, so I couldn’t really gather the UUIDs in order to search through configurations etc. The master refused to talk to the pool server, and the pool server with the broken (and afterwards replaced NIC) refused to let me change anything, because that should be done on the master. Catch 22.

I consulted the official support forum, but nobody knew an answer there either. I’m sure there is a way to change it easily. After all it’s a Linux box with a modified Xen, but still not an unaccessible blackbox. Hang on… actually it felt a bit like that. I would like to think that Citrix certainly knows an easy solution, but as I’m not paying thousands of Pounds for a product, which is almost entirely based on free software, they of course kept quiet. (The bloody toolstack, which complicated things, is their own development, by the way.)

End of that experience was that I had to remove the server from the pool (XenServer would then wipe the box, so you can’t re-join the pool later, either… awesome). After a clean setup and restoring all the VMs from previously created snapshots, the machine was finally able to join the pool. That was 6 hours after the NIC broke. Fortunately all VMs have an identical twin running on another machine, so it didn’t cause downtime (except a few minor hiccups while I was fiddling about with network settings). Otherwise all websites/applications would have been offline for 6 hours.

Without the XenServer toolstack, I could have resolved the issue within 10 minutes, which includes all of the steps mentioned earlier (what I would have expected).

I learned my lesson from it. As live-migration of VMs isn’t really necessary in most cases (my customers’ applications don’t benefit from it), it’s actually better to not form pools of your servers. Disconnected standalone servers are a lot easier to maintain and you don’t risk side-effects with pool members, because there aren’t any. The only real downside is that VLANs need to be configured individually on each server. Same applies to shared resources (NAS etc). But that’s fine.

Another almost unbelievable example is deleting snapshots. I create them all the time, because if something goes wrong, or someone breaks a VM setup, you want to be able to roll back to a previous version. Snapshots are one of the biggest advantages of virtualisation. A whole VM can be brought back to an older state within seconds. Or you can export it and reimport it elsewhere, clone another instance from it, work there, swing later. Anyways, if you use that feature often, it fills your disk (even the huge disks you get nowadays). So you regularly delete them and get your space back. Right? Nope, wrong. With XenServer you may or may not get your space back. When your monitoring tells you that you are running out of disk space, although you haven’t done anything but rotating snapshots in a while, you scratch your head in disbelief. Well, at least I did. Unfortunately, the official documentation confirms my observations. When I first read that reclaiming space causes downtime, I wasn’t sure if laughing or crying was the best course of action.

In a production environment, you can’t just go ahead and suspend VMs just to get space back. Even if you only reduce performance (without causing downtimes, as we’re running twins of everything), you need to make affected customers aware of it. And how do you explain that? “Sorry, Sir, I need to suspend your service, because I need to delete old snapshots.” They’ll think you’re taking the piss.

Again, this “feature” is brought to you by Citrix’s toolstack, not Xen. If I decide to delete an LVM-based snapshot of a running VM on Xen, I can do that any time. No need to suspend anything or to manually reclaim free space afterwards.

My favourite subject is VLANs. I don’t know how many hours I’ve wasted trying to find what I did wrong, just to figure out in the end that it was not my fault… Citrix apparently manipulated the bridge code and never really tested it. You have to actually install ebtables (iptables for bridges, if you will) to work around that issue. I observed exactly the same thing as the poster there, and many others did, too. Their forums are full of problems related to VLANs and NIC bonding. Problems get worse with two NICs. VLANs may work out-of-the-box on both, only one, or none of the NICs. Apparently it depends on the used NIC (well, I’m assuming here that nobody uses old NICs without VLAN support any more nowadays), which of the NIC is management interface, and a couple of other factors like weather, mood etc.

Once you know about the workaround mentioned earlier, you can solve it. But now, when you update your XenServer version, you can’t rely on Citrix. They might just remove the required kernel modules so that ebtables wouldn’t work any more. Sounds unlikely? Well, reality is that ebtables did work until XenServer version 5.5, but in 5.6 the kernel support was removed (see last post here). To fix it, you end up downloading the XenServer SDK (which includes all the open source bits they are using) and recompile the kernel yourself.

I won’t go deeper into this subjcet, but there are several issues with bonded NICs as well. And the management interface can never be on a tagged VLAN. All those are restrictions/problems solely related to Citrix’s stuff. Linux itself lets you create any combination of bonds and VLANs on as many interfaces as you want to. Unfortunately, you need to unlearn all about Linux network configuration, because if you try applying your knowledge, XenServer will overwrite your configuration as soon as you reboot (best case) or use its API or Windows client to manage NICs/VLANs.

I could go on and on and on. There are many other quirks like being unable to shutdown a VM when for some reason it can’t attach to a VNC console (but keeps trying, although you absolutely don’t need a console to shut it down); having a “force” option for many commands, which is useless, because it doesn’t force anything; being unable to remove stale shared storage; having to work around limitations which would for example disallow you to build a pool with an i7 920 and an i7 930 server; and quite a few more, which are of minor relevance in a production environment.

Don’t get me wrong. If you dig deep enough, you will find problems in any similarly complex software. And Citrix’s XenServer is not a bad product at all. Much of the functionality like live-migration isn’t available in VMware’s free version ESXi, and said free version doesn’t run on top of CentOS but on a custom Linux, which officially you can’t access via SSH (there are ways though, but you can’t expect any support at all). Also, XenServer’s GUI is self-explanatory and easy to use — and certainly one of the main reasons for using XenServer, because whoever is going to use it after you set it up for them, they won’t have many problems getting started.

However, if you don’t have lesser knowledgeable people using it later, and if you don’t mind going the extra mile, you probably get most flexibility and reliability if you set up Xen instead (the vanilla or “real” one, not XenServer). XenServer doesn’t really provide any additional functionality, which isn’t available in Xen. (Some people even say the opposite is true, and you only get full Xen functionality if you purchase XenServer’s extra licenses; I wouldn’t go that far.) It does add convenience with its GUI and toolstack though, which you’d otherwise have to implement yourself — snapshots, shared storage use, starting up any type of guest OS etc. Most of those things aren’t exactly rocket science; only a few are a bit more tricky. But you can script/automate them as you please and you don’t need to expect any bad surprises caused by 3rd parties.

For example, I disabled Xen’s bridging code (by commenting out a single line in their scripts) and do the whole network configuration with standard OS tools, keeping it independent and consistent for future updates. (More details here.) Snapshots are easy enough to do with LVM, too. Live-migration I haven’t tested yet, but it doesn’t look too difficult to do either. (We don’t really need that feature here anyway)

What I’ve struggled with was getting different OS running, namely FreeBSD. But now that I have sorted that out, I can easily clone and fork more FreeBSD VMs on the vanilla Xen machines. Hence, Citrix XenServer isn’t providing any benefits there either.

As you can see (and as the title suggests), I’m considerably fed up with XenServer’s quirks; some of them are too huge to accept them in production environments. Consequently, we’re going to “migrate” back to Xen, where we can. (Admittedly, in some environments we won’t be able to do that for another year or so.)

Once you’ve worked out how XenServer stores VM backups (yep, they did their own thing there too, and the format is really stupid), it’s not too difficult to convert them. I’ve done that for both CentOS and FreeBSD XenServer images. They run smoothly on vanilla Xen after converting them back.

Once again the “keep it simple” motto wins. Additional toolstacks and bloat cause more problems than necessary, and the manufacturer turns out to be the only one benefitting from it — as often is the case. So long, XenServer — Hello Xen!

(Update: Only three hours after I published this, one of our XenServers started refusing to create new VMs from templates…)

(Update 2: It’s cursed. Yesterday I was all of a sudden unable to attach any block devices, hence I was unable to start new VMs, reboot existing ones, or increase storage. I’m not the only one, who faces that problem and does not get any help from the experts at Citrix.)

(Update 3, Aug 25th: Done. Last weekend we’ve transformed the last remaining XenServers to vanilla Xen. Thanks to the twin-design, this went through without any downtimes whatsoever; was a major piece of work though, but certainly worth it. Chapter closed. )

Admittedly the learning curve for both is quiet steep, and everyone’s well-advised to spend some serious time evaluating them. When I started looking into EC2, I only had a rough idea of all the services they offer. I was quite overwhelmed, how many related services EC2 (or more precisely AWS) entails:

• EC2, the cloud, which runs your instances (also known as virtual machines, Xen based) in one out of four regions (US East/West, EU, APAC) and one out of two-four availability zones in each region
• Elastic Load Balancing (ELB), giving you the opportunity to spread load across instances, obviously
• Elastic IPs, allowing you to assign (and re-assign) static IPs to instances of your choice
• Simple Storage (S3), which guarantees replication of your stored data in three different locations, enabling it to survive an outage of two entire data centres (or one data centre, if you opt-in for the “reduced redundancy” option, which is a little bit cheaper — you can choose that for every file stored individually)
• EBS (Elastic Block Storage), enabling you to create RAID-backed volumes of any size and attach them to any of your EC2 instances; on top of that you can create snapshots (which are internally stored on S3) within seconds
• RDS (Relational Database Service), basically a MySQL offering, in either single, single/hot-standby, master/slave, or master/multi-slave setups, with nodes spread across different availability zones
• CloudWatch, which entails monitoring facilities for most of the services
• CloudFront, a multi-region CDN-like service
• SimpleDB, Map/Reduce
• Route 53 DNS services (beta)
• DevPay, Flexible Payments

All these services have one thing in common: They can be managed entirely via different APIs and command line tools. There’s nothing which you can’t automate, if you spend some time and effort to actually understand how it all fits together! It’s certainly very overwhelming in the beginning, and Amazon clearly doesn’t target customers who might want to fire up one or two instances and that’s it. It’s way too complex for that. And it requires an entirely different approach, for example an instance and all its data is lost when you terminate it. And all resources are very dynamic — for most simple use-cases too dynamic (you don’t know which IP or hostname your instance will have; most provided OS images won’t suit your needs, so you’ll need to build your own). But if you are interested in creating environments for your applications, which come with both high availability and scalability, then EC2 is definitely worth a shot. Amazon gives you the bullet-proof and battle-proven infrastructure and tools — you need to decide and find a way how to use them for your requirements.

Amazon offer the AWS console for very basic management of your resources. Very basic. You’ll soon find out that it can’t offer things, which you really will need:

• creating a snapshot, which you can use to boot another instance from (or as a backup to start the same instance again, when it fails)
• setting triggers for the CloudWatch monitoring (or alarms as they call it in their API)
• bundling your instance (or parts of it) and backing up on S3
• moving instances between availability zones
• configuring the RDS MySQL server
• and many more things

All of these things can be done via API (in Java, PHP, and other languages, or via command line tools, which can all be downloaded from Amazon). Some of them are trivial, most are not. Flexibility takes its toll. Consequently, you should be prepared to spend some time tailoring your own toolset. There are some 3rd party offers out there (notably the best on is s3cmd, which allows rsync-style file transfers between instance and S3 buckets). They may or may not suit your needs.

I’ve spent the last two weeks creating my own toolset. With very simple commands I can now build fully bootable AMI images for different Linux setups both in 32bit or 64bit (EC2 instance types differ in terms of architecture!), create bootable snapshots from running instances, detect instance failure and restart from the most recent snapshot (including re-assigning the elastic IP), set tags and other information/attributes on all sorts of resource types, create volumes (empty or from snapshot) and attach them to instances, hook instances into a load balancer, read all relevant CloudWatch metrics and feed them into RRD graphs, clone instances on-the-fly, launch any number of clones, manage security groups and keypairs etc.  Basically everything the AWS console can plus a few necessary features on top of that — with a single shell command and no more than 2-3 parameters each. I’m not exactly a developer and started doing this merely as a proof of concept (but then went further than originally intended to). If I can do that, some of you bright-minded developers can do a lot better for sure

This was AWS management covered. But how about managing the actual instances (their OS internals)? What if, for example, you want to deploy a web application on four identical, load-balanced nodes?  Should I create a dedicated image for that (not too difficult with my toolset)? Or would it be better to have a look into Puppet at last? I went for the latter. I’ve got customers on my own clusters outside of EC2 (mostly based on Citrix XenServer), and that environment is growing continuously. It’s about time that I simplified management there as well.

Consequently, I decided to take my EC2 proof of concept another step further. After getting acquainted to Puppet, I’ve deployed it on a playground-style bunch of EC2 instances and told it to install/configure various things. The language structure really gave me hard times in the beginning, but once you get used to it, you can almost write it down as you think.

The next thing I wanted to achieve was that puppet connects to the puppet master as soon as the instance is started. There were some obstacles in the way, though: AWS assigns hostnames dynamically, but your puppet master would need to know that hostname in order to sign the certificate used for communication between both. Catch 22 situation. Resolved by writing a tiny web service which allows the instance to figure out and set the hostname I assigned (and dynamically added to a DNS server as well) rather than using Amazon’s one. This happens during startup just after the network interface comes up, so that all running services use the correct hostname. Puppet then takes over at the end of the first startup of the instance and installs/configures as told by the puppet master. This way you can fire up a whole cluster, hook it into the load balancer, and are ready to go live in just under three minutes. Fully automated. And the monitoring mentioned earlier would pick up metrics via CloudWatch instantly.

I’ve heard it all in theory before. However, I wanted to see my own working proof of concept for a few things (and some others, which are still in progress). I’m pretty amazed actually, how much flexibility and reliability (often a contradiction in terms) AWS offers. You just have to embrace a slightly different model of implementing things (you’ll like the term “ephemeral”, which Amazon have chosen for a reason!).

So now the next question would be: How much does it cost. Is it really saving costs as many people state? Frankly, I don’t know yet. It may do. Surely, it reduces upfront costs, as Amazon won’t charge any setup or recurring fees, unless you opt-in for their “Reserved Instance” schemes, which are actually significantly cheaper in the long run. For example, a Micro instance (640 MB Ram, 1.7GHz Xeon; the smallest instance type) would usually be charged at US$ 0.025 per hour, which equals to US$ 219 per year if running full-time. If you commit to a year, paying US$ 54 one-off, your hourly rate is reduced to US$ 0.01, which together equals to yearly costs of US$ 117.60 or less than US$ 10 per month! That’s a smashing 50% discount almost. Higher discounts possible, if you can commit to 3 years.

However, the pricing is somewhat difficult to decipher and costs impossible to predict. I don’t actually know yet, how many IOPs (I/O operations) my EBS volumes and snapshots will generate. I can’t exactly tell how much S3 storage I will use. Also, I don’t know exactly what to expect on the inter-availability-zone traffic scale. Or the RDS (MySQL) read/write operations. Surely, previous monitoring gives me very rough estimates, but not good enough to make an educated guess as to what costs to expect on EC2. I will have to keep an eye on that over the next weeks and months, and also find some tools to get all these figures from the usage reports (CSV or XML files, downloadable from Amazon). At least you can see how your usage translates into actual costs for the current billing period, updated every few hours. So the costs wouldn’t hit you as a big surprise

On the plus side, you never pay for any over-capacity, which you would need to account for, if you built everything in-house. When you build infrastructure like that on your own, there are different things, which scale more or less dynamically (if you’ve got 10 servers already, buying two more doesn’t do a harm). But you’ve also got devices where upfront costs are enormous, because you buy them from the point of view, of what you might need in the foreseeable future, not what you do need at this very moment. Storage devices are a good example. A chassis from NetApp with only a few drives cost you an arm and a leg; then you can scale it a bit for a reasonable price; and then you’ll need another one sooner or later. But you always end up paying for more than you actually use at any given point. Same for networking devices.

Clouds like EC2 take those massive entry-costs from you (and your customers), which saves painful budget discussions. They’ve got a brilliant, scalable infrastructure, and one would be bold to assume that you could build anything better at a reasonable price (also take availability in distinct and independent data centres into account!). Now that they’ve got their PCI DSS certification, one of the biggest remaining concerns (what about data security in a proprietary, shared environment?) for many customers is gone, too.

I’ll go through the other proofs of concept on my list, see how usage translates into actual costs over time, and may then be able to add some very interesting offers to my company’s portfolio. Stay tuned

Surely, the cloud is not the solution to all problems (although it’s commonly propagated as that), but with decent automation and tools it can improve or at least add value to a variety of services.

When I moved in a year ago, there wasn’t a broadband connection in this house, but there was a phone line — with BT apparently. So I went for the option, which I hoped would be the quickest (in terms of getting connected): BT. 8meg sounded ok to me, and I didn’t want to use 3G all the time (although I had a lot of traffic allowance on my contract).

We were in fact online after a couple of days. However, we did not get 8meg. We were around 6meg first, with a rather ridiculous upstream speed:

But, as we all know BT, things never get better. They only get worse… This year we had “impressive” results, especially in the evenings:

This picture might look like an exception, but it’s not. Towards the end of the contract things got even worse:

I’d like to point out again, that this connection was supposed to be a 8meg connection!

Even my T-Mobile 3G connection could compete with that:

We were told that our line quality is too poor to offer higher speeds. And the exchange was too far away, they said. In fact it’s only about 300m down the street! Plus, one might wonder why we got 6meg downstream half a year earlier then? Why did it constantly get worse, especially in the evenings? Right, because the uplinks from the exchange onwards are blocked. They connect more customers than they actually can handle. Should introduce Congestion Charge there, seriously! So, probably it doesn’t surprise anyone that we did not stay with BT.

Virgin Media’s fibre optic offers were very appealing. Our neighbours next door have it. Our neighbours on the other side next door can have it. And even in the same house, our neighbours upstairs can have it. That’s what the online availability checker said and still says. Guess what? Right, we cannot have it!

I thought a support call might actually help discover an error in their database, and in fact we can have it, too. Well, I thought. Obviously every man-hour is way more expensive than the monthly fibre optic rental customers are being charged. So they showed little to no effort, and told me that I can’t have it, because that would be technically impossible. Why? The cable is out there, and my neighbours upstairs can have it. Why would it be technically impossible? The sophisticated answer was: I don’t know. The system shows it’s impossible and wont change in due course.

A second call, hoping to talk to someone who’s a little bit brighter, didn’t make a difference. Other words, same meaning.

People recommended Be* and reckoned that they had a good customer support, too. Interestingly they estimated a possible line speed of 19meg!  Wow! To be fair, I would have been happy with anything beyond 4meg, which wouldn’t drop in the evenings.

Only one week after we decided to switch, we were connected with Be*. There was only a 3 or 4 hours gap on the day they switched the line. While I was working from home, I could fill that gap with T-Mobile 3G.

One of the first things I did after we got connected was, of course, to verify the line speed with Be*. Here’s the first result:

Yay! But…wait. Didn’t they say 19meg downstream and 2.5meg upstream? So we only got half of the promised speed. Hmm. Of course, still way better than BT ever was — on the same physical phone line! However, we are humans… we always want more, especially when someone promises to give more. So I quickly changed my mind and decided not to be happy to have 4meg or more. That’s probably because I somehow expected that the line wasn’t capable of offering more than the 6meg we initially got from BT. But as it was, I became greedy

After a couple of support tickets and two weeks later, the best results we ever got (relatively stable) were these:

Be*’s support turned out to be good with simple things, and total failure with difficult things. Arguable if support is the right term then. Actually their user forums were more helpful than their paid staff. Ping here, traceroute there, connect via Ethernet rather than wireless… Also they didn’t really read the whole trail of the ticket. Just the last message. Which, obviously, resulted in the same questions being asked over and over again — and a solution being severely delayed.

The funny thing was that the BeBox (the provided DSL router) synced with different speeds on every single re-connect. Plus, it started to reboot randomly, dropping the DSL connection for 5 minutes each. Sometimes 4-5 times a day (probably more often, but we didn’t notice). How would a ping or traceroute help to solve this? (Yes, I have been asked a couple of times to provide pings and traceroutes to google.com and the BBC!)

Anyway, let’s continue…

Be* offers three different DSL profiles (sync setups), which customers can choose from:

• optimised for speed
• normal
• optimised for reliability

Plus, you can opt in for the fast path option, which gives way faster round trip times at the cost of reduced error correction.

The BeBox was using the “normal” profile without fast path. We did try the “optimised for speed” option, but that made things worse. The disconnects really bothered me, but I didn’t want to reduce the speed further. So the picture above shows the maximum we could get through the line using a BeBox, and accepting a couple of disconnections a day. I wasn’t impressed.

In different forums I found people complaining about the BeBox, describing similar symptoms. So I thought: Let’s just try another router, for whatever it’s worth.

I ordered the D-Link DSL-2640B (£50). When it arrived, it took me about 30 minutes to get it running (20 minutes to get the DHCP lease renewed, 10 minutes configuration and reboot).  And here are the very first results (still with Be*’s “normal” profile, and without fast path):

Did you notice? More than 3meg more downstream with exactly the same line and cabling, just a replaced router. And this was an average speed test result. (By the way: I did not only use speedtest.net with their Maidenhead server; I double-checked the results with other tests, too — all of the reported results in this article)

But it gets even more interesting: As the D-Link apparently can deal way better with a below-average line quality, I wondered if I could push it a bit. Today I asked Be* to switch to “optimised for speed” and activate fast path on my line. And now watch this:

To digest this article… On the same physical line, I got:

• 6meg initally with BT
• dropping down to 2meg with BT
• 11meg with Be* and their BeBox
• 16.5meg with Be*, optimised settings and a D-Link router

Using Be* with a third-party router can result in 10.5meg more bandwidth than BT said would be possible on that phone line, and even 5.5meg (50%!) more than Be*’s BeBox can achieve on that line.

The D-Link router didn’t show any uncorrectable errors so far. No disconnects. No other unexpected problems. But very good performance!

Why the heck do the ISPs bundle crap hardware with their offers, causing unnecessary support inquiries? And why do the ISPs not have support staff in place, who are actually capable of dealing with the increased support load then?

Or in other words: Why does the customer have to spend lots of time and a bit of extra money to figure out and solve the issues on their own, while all they requested was to get what they actually pay for?

[ Update: I just realised that Be* switched the line back to the normal profile without fast path active. Sync speed and throughput went immediately back down to the old values. Let's see how long it takes to get my preferred settings activated (and hopefully persisted!) again. I'm getting slightly mad at them. And the option on their "website" (quotation marks on purpose -- see youself!), where I could in theory choose the setting myself, is broken, too. So I have to wait for their support to do it... ]

[ Update 2: About one hour later, I'm back to the desired settings... Let's see for how long. They claim it has been changed on the member portal. I wish I could do that, but firstly it's broken there, and secondly I haven't touched it at all... Funny people. ]

First of all, questions like this, which do not tell anything about the author’s aims and intentions, are not answerable. One could as well ask: Ferrari or Landrover? I’d suggest taking the Ferrari for the next cross-country rally, whereas the Landrover is definitely the best choice for the F1 track. Anyway, you got my point.

This blog post has potential for flamewars between the lovers of BSD and Linux, and also between lovers of either of the Linux distributions. So let me emphasize that this is my personal opinion.

Let me kick off with two certainly arguable statements and take it from there:

• Production environment: The operating system of your choice should be the one, which you are most comfortable administering, because it’s your job to secure it to the best of your knowledge and solve upcoming issues within the least possible amount of time and effort.
• Experimental environment: Do whatever you want to. Experimental environments are meant to gain more knowledge, experience or compare it with other environments.

In this context, let’s be clear about this: Any server that is accessible from any other untrusted machine (aka Internet), is a production environment! Why so? Because it could easily be turned into a threat to others (if not secured properly), which can cause trouble with your ISP or with third parties, which leads to costs, and in the worst case lawsuits! This means: Although you run it for your own pleasure, you have to ensure that your pleasure does not become a nuisance to others — be it by your mistake or by third parties taking over your server. Should be common sense, but apparently it’s not.

Now that we’ve understood that the playground approach is misplaced in a server environment, you may want agree with my previous statements.

“I hear you, but which Linux/BSD/Unix is the best for which aims?” Let me first briefly explain how things have evolved and why a FreeBSD user will have problems recommending any Linux distribution.

Unlike any Linux distribution, which strictly speaking is merely the kernel bundled with a bunch of (mostly) GNU tools and programs, FreeBSD is a real operating system, where all core elements are maintained by a central “authority”, the FreeBSD Project (which is funded by donations collected by the FreeBSD Foundation). That ensures a high level of integrity and as a result stability. FreeBSD (like NetBSD) is a fork of the original BSD by the Berkeley University, which was derived from AT&T Unix. Nowadays you find three major BSDs out there: FreeBSD, NetBSD, and OpenBSD (which was forked from NetBSD). They are maintained by their core teams, and cross-port various functionality whenever suitable (e.g. OpenBSD’s packet filter pf). When you install any of these BSD’s base, you will end up with a working operating system and all core tools needed to administer it.

When you install any of the approximately 250 different Linux distributions out there, you more precisely install a third-party bootloader, the Linux kernel, and a whole bunch of third-party (GNU) tools and software. What exactly you end up with, depends on the taste and policies of the distributors. It should be easy to understand that a distribution which focuses on including the latest drivers and software in every release, cannot be as stable as a distribution with a rather long release cycle that has got a big number of enterprise-level users. Essentially they are all the same, but the collection of software and tools (and their branding and look&feel) differs. As various GNU projects have got a lot of cross-dependencies (e.g. PHP with GD, ImageMagick, MySQL, to mention a popular one), it is a tedious and time-consuming task to bundle the right versions with each other in order to get a stable system.

Or in other words: A Linux distributor has to ensure that their selection of third-party software form a stable system, whereas the major BSD derivates maintain the core system themselves. In the BSD world, third-party software isn’t part of the core functionality. Hence BSD doesn’t depend on the good will of other software projects. However, you can of course get a lot of third-party software, too: The portstree (in FreeBSD for example), currently contains over 20,000 different programs, carefully selected and tested, and linked against other ports and/or the core libraries. As the latter are provided by the BSD maintainers, you can be sure to have a solid foundation.

If you look at SELinux, it was a rather chaotic uncoordinated situation in the beginning: SELinux was developed and maintained by the NSA, and was not part of the kernel initially, but you could compile it as a kernel module (don’t get me started on kernel modules on a server). The tools to actually use it are part of the GNU coreutils package — third party software, strictly speaking. When SELinux reached a stable status and was supported by the Linux kernel, some distributors decided to include and activate it by default (Fedora, RHEL, and CentOS), while others didn’t make use of it at all (Debian, Ubuntu). So security was a matter of the distributor’s taste. That happens when there’s no central “authority” which ensures continuity, and coordinates kernel (and related) development. A sad result was that people didn’t want to get used to SELinux, because it wasn’t (and still isn’t) accepted as a standard and must-have. Even nowadays you read recommendations like “use ‘setenforce 0′”, which effectively switches SELinux restrictions and its security improvements off! As far as I know, only RHEL and CentOS install and activate SELinux and its utilities by default. They are also the only mainstream Linux distributions which activate the iptables firewall by default, and apply a restrictive ruleset, by the way.

You’ll still even find Linux distributions, which allegedly target the server market, without SELinux utilities installed. How can you ignore huge security enhancements in a server environment? Ah right, the distributor has got a different taste and would probably add no other security tools.

Apologies for my sarcasm. Linux is not all bad, but you must not expect any distribution to be as rock-solid as any of the three main BSDs. Let’s check out which Linux is the least of all evil

The first commercial distribution back in the early 90′s was Slackware, which nowadays is only being used on a minority of Linux-based servers. Slackware is sort of considered geeky.

A couple of years ago, the big players were RedHat Linux in the English-speaking countries, and SuSE in the German-speaking areas. That has changed. RedHat Linux for the commodity market does not exist any more (it is now the community-maintained Fedora Linux, supported by RedHat). RedHat’s own Linux distribution is RedHat Enterprise Linux, which obviously targets enterprise-level customers, who are willing to pay for licenses and professional support. For those who don’t, CentOS as a de-branded RHEL copy has become more and more popular. It claims 100% binary compatibility with RHEL, without asking for license fees, and without offering professional paid support. The target group for both RHEL and CentOS are enterprises and server installations, whereas Fedora targets the desktop market.

Especially in Germany, Debian Linux is also widely used. The easy package management with apt-get certainly played an important role in its success. Fedora introduced yum to make RPM package management as easy. (Open)SuSE seems to lose market share. YaST as their package manager could be one reason.

Ubuntu was started as a Debian derivate just a few years ago, and initially aimed the desktop market trying to keep up with current hardware drivers and new features. Apart from its LTS (long term support) versions, which are being released once every two years, the life cycle of the half-year releases is very short. However, Ubuntu has experienced great success and played an important role in making Linux a widely accepted desktop operating system. In my opinion it is not the best choice for server installations where robustness is more important than introducing the latest features and device drivers, though. It also lacks SELinux utilities in its default installation.

For the tough cookies, there’s also Gentoo Linux, which covers kind of a niche market: people who believe in stability by compiling everything from scratch (and most obviously took BSD as an example), but who are reluctant to leave the Linux terrain towards BSD. Why am I saying this? Simply because compiling from scratch is rather not suitable for newbies, but all the effort still doesn’t provide a BSD level of stability in a Linux environment as too many bits and pieces are actually third-party software.

Personally, although I used to prefer Debian (before SELinux became de facto standard), I cannot take that distribution seriously any more. They made a terrible mistake when they “patched” the OpenSSL library, turning all generated keys and certificates built with them into garbage (or what do we call keys and certs which are created with a predictable random generator?). In my opinion, it shows pretty well why a more centralised approach of maintaining core components is better. There’s a thin line between diversity and mess. When distributors start patching core components just like that, rather than contributing code to the upstream projects, the diversity will soon equal mess — and introduce absolutely unnecessary distribution-related security flaws.

In my personal opinion, CentOS is the Linux distribution for a server setup (or RHEL for those who rely on professional support), whereas a desktop or laptop user’s best bet would be Ubuntu (if you can live with a short release cycle and are happy to update your whole system often) or Fedora.

However, I do prefer FreeBSD for servers (as you could easily tell after reading all this).  And on a desktop/laptop, MacOS is my favourite. (I know that paying for solid software is political incorrect nowadays, but at least it has a reliable — FreeBSD/Darwin — foundation!)

Ok, everyone has Facebook (at least in English speaking countries people would rather ask if you are on Facebook than what your phone number is). And then there’s Twitter, another way of keeping friends or customers or whoever informed about what’s going on. Plus, you need to share you favourite URLs with digg, del.icio.us, or any other social bookmarking service of your choice. And of course, you need your own blog! And a Flickr account for your photos. Did I cover all of the services a modern Web 2.0 person has to have? Probably not. But now the core question: How do you manage to keep everything up 2 date? I mean as someone who’s already got a full-time job…

Comments, suggestions, URL appreciated!

But as I took a closer view at it recently, it turned out to be a pretty simple task. Only people with loads of domain names and/or restricted nameserver control might run into problems.

In a nutshell, you only need to do this in order to get SPF running on your own server and to tell other mail servers how to deal with your domain name:

• add a policy daemon to your MTA (e.g. postfix-policyd-spf) — pretty easy, really!
• add SPF/TXT records to your zonefiles as described here

That’s it, honestly. Assuming, you have successfully finished the two tasks, your mail server will block all mails that originate from domains, which have SPF records set and been delivered through other than the allowed hosts. And your domains will be protected from being misused on all other mail servers which use SPF as well.

Example:

example.com.       IN MX 10  mail
mail.example.com.  IN TXT "v=spf1 mx -all
mail.example.com.  IN A 1.2.3.4

Now, mails from johndoe@example.com must be delivered through the mailserver mail.example.com. All mailservers which implement SPF will refuse to accept mails from other than that server. Especially the big players like Googlemail do make use of SPF. Although some don’t block mails, they at least add a telling header which makes spam-filtering easier:

Received-SPF: fail (google.com: domain of mail@***.co.uk does not designate
85.***.***.*** as permitted sender) client-ip=85.***.***.***
Authentication-Results: mx.google.com; spf=hardfail (google.com: domain of
mail@***.co.uk does not designate 85.***.***.*** as permitted sender)
smtp.mail=mail@***.co.uk

So what are you waiting for? The more people make SPF mandatory on their servers, the better is its protection against SPAM.

This was how I got my first Mac OS based computer. It did not take long to make me love it, because the difference between Mac OS and other operating systems is, that they kept an eye on detail. They built an OS which clearly works and seldomly causes any hassle. Mac OS combines a great UI with an incredibly good usuability and a solid and stable foundation: Darwin/BSD. Since I got my MacBook Pro, it never freezed, it never crashed and it never did anything unexpected. Can Windows do that? And there was no problem getting parts of the hardware to work properly within a minimum amount of time. Can Linux do that?

To cut a long story short: I am not a freak any more who is willing to spend hours on setting up his computer and getting peripherials to work. I expect my laptop to work out of the box. And I really do hate unexpected behaviour. Mac OS has convinced me as a desktop/laptop operating system.

And what about the hardware? Well, it is at least as convincing as the OS! Of course, you can get the same piece of hardware cheaper, if you only look at the technical data:

• Intel Core2Duo 2×2.4 GHz
• 2 GB RAM
• 200 GB SATA HDD Fujitsu connected to Intel ICH8
• Broadcom WLAN, Ethernet (1Gbit), Bluetooth 2.1
• 15.4″ Widescreen TFT (1440x900px)
• DL-DVD/CD burner
• Firewire 400, Firewire 800
• 2x USB 2.0
• Audio in/out
• DVI connector

That’s certainly a rock-solid equipment, but nothing which justifies a price of £1,299. To be honest, technical data is only one part of the story. A good laptop is more than a collection of good components. And this is why a MacBook Pro is better than most other laptops:

• Battery lifetime: using office applications and terminal windows, I can run the MacBook Pro for more than 5 hours without power supply!
• Trackpad: it is precise and huge enough; together with the two finger scrolling and zooming functionality I hardly ever need a mouse
• Magnetic power plug: Did you ever pull the power cable incidentally? Don’t worry, because the magnetic plug is safe
• Air flow: There’s no risk of over-heating the laptop when putting it on a soft surface, because the air flow cannot be covered
• Robust case
• Great keyboard: except from getting used to Apple’s special keys (which only applies to new Apple users), they keyboard is excellent
• Reasonable sound quality: Compared to other laptops, the speakers are quite ok. Of course it cannot compete with a home stereo
• Excellent display (I use the matt one, because I don’t like makeup mirrors)
• last but not least the aforementioned Mac OS X (“Leopard”)

So, yes the MacBook Pro is expensive. But it is clearly worth its price! After having used it for almost three months, I can say: I love it!