Admittedly the learning curve for both is quiet steep, and everyone’s well-advised to spend some serious time evaluating them. When I started looking into EC2, I only had a rough idea of all the services they offer. I was quite overwhelmed, how many related services EC2 (or more precisely AWS) entails:

• EC2, the cloud, which runs your instances (also known as virtual machines, Xen based) in one out of four regions (US East/West, EU, APAC) and one out of two-four availability zones in each region
• Elastic Load Balancing (ELB), giving you the opportunity to spread load across instances, obviously
• Elastic IPs, allowing you to assign (and re-assign) static IPs to instances of your choice
• Simple Storage (S3), which guarantees replication of your stored data in three different locations, enabling it to survive an outage of two entire data centres (or one data centre, if you opt-in for the “reduced redundancy” option, which is a little bit cheaper — you can choose that for every file stored individually)
• EBS (Elastic Block Storage), enabling you to create RAID-backed volumes of any size and attach them to any of your EC2 instances; on top of that you can create snapshots (which are internally stored on S3) within seconds
• RDS (Relational Database Service), basically a MySQL offering, in either single, single/hot-standby, master/slave, or master/multi-slave setups, with nodes spread across different availability zones
• CloudWatch, which entails monitoring facilities for most of the services
• CloudFront, a multi-region CDN-like service
• SimpleDB, Map/Reduce
• Route 53 DNS services (beta)
• DevPay, Flexible Payments

All these services have one thing in common: They can be managed entirely via different APIs and command line tools. There’s nothing which you can’t automate, if you spend some time and effort to actually understand how it all fits together! It’s certainly very overwhelming in the beginning, and Amazon clearly doesn’t target customers who might want to fire up one or two instances and that’s it. It’s way too complex for that. And it requires an entirely different approach, for example an instance and all its data is lost when you terminate it. And all resources are very dynamic — for most simple use-cases too dynamic (you don’t know which IP or hostname your instance will have; most provided OS images won’t suit your needs, so you’ll need to build your own). But if you are interested in creating environments for your applications, which come with both high availability and scalability, then EC2 is definitely worth a shot. Amazon gives you the bullet-proof and battle-proven infrastructure and tools — you need to decide and find a way how to use them for your requirements.

Amazon offer the AWS console for very basic management of your resources. Very basic. You’ll soon find out that it can’t offer things, which you really will need:

• creating a snapshot, which you can use to boot another instance from (or as a backup to start the same instance again, when it fails)
• setting triggers for the CloudWatch monitoring (or alarms as they call it in their API)
• bundling your instance (or parts of it) and backing up on S3
• moving instances between availability zones
• configuring the RDS MySQL server
• and many more things

All of these things can be done via API (in Java, PHP, and other languages, or via command line tools, which can all be downloaded from Amazon). Some of them are trivial, most are not. Flexibility takes its toll. Consequently, you should be prepared to spend some time tailoring your own toolset. There are some 3rd party offers out there (notably the best on is s3cmd, which allows rsync-style file transfers between instance and S3 buckets). They may or may not suit your needs.

I’ve spent the last two weeks creating my own toolset. With very simple commands I can now build fully bootable AMI images for different Linux setups both in 32bit or 64bit (EC2 instance types differ in terms of architecture!), create bootable snapshots from running instances, detect instance failure and restart from the most recent snapshot (including re-assigning the elastic IP), set tags and other information/attributes on all sorts of resource types, create volumes (empty or from snapshot) and attach them to instances, hook instances into a load balancer, read all relevant CloudWatch metrics and feed them into RRD graphs, clone instances on-the-fly, launch any number of clones, manage security groups and keypairs etc.  Basically everything the AWS console can plus a few necessary features on top of that — with a single shell command and no more than 2-3 parameters each. I’m not exactly a developer and started doing this merely as a proof of concept (but then went further than originally intended to). If I can do that, some of you bright-minded developers can do a lot better for sure

This was AWS management covered. But how about managing the actual instances (their OS internals)? What if, for example, you want to deploy a web application on four identical, load-balanced nodes?  Should I create a dedicated image for that (not too difficult with my toolset)? Or would it be better to have a look into Puppet at last? I went for the latter. I’ve got customers on my own clusters outside of EC2 (mostly based on Citrix XenServer), and that environment is growing continuously. It’s about time that I simplified management there as well.

Consequently, I decided to take my EC2 proof of concept another step further. After getting acquainted to Puppet, I’ve deployed it on a playground-style bunch of EC2 instances and told it to install/configure various things. The language structure really gave me hard times in the beginning, but once you get used to it, you can almost write it down as you think.

The next thing I wanted to achieve was that puppet connects to the puppet master as soon as the instance is started. There were some obstacles in the way, though: AWS assigns hostnames dynamically, but your puppet master would need to know that hostname in order to sign the certificate used for communication between both. Catch 22 situation. Resolved by writing a tiny web service which allows the instance to figure out and set the hostname I assigned (and dynamically added to a DNS server as well) rather than using Amazon’s one. This happens during startup just after the network interface comes up, so that all running services use the correct hostname. Puppet then takes over at the end of the first startup of the instance and installs/configures as told by the puppet master. This way you can fire up a whole cluster, hook it into the load balancer, and are ready to go live in just under three minutes. Fully automated. And the monitoring mentioned earlier would pick up metrics via CloudWatch instantly.

I’ve heard it all in theory before. However, I wanted to see my own working proof of concept for a few things (and some others, which are still in progress). I’m pretty amazed actually, how much flexibility and reliability (often a contradiction in terms) AWS offers. You just have to embrace a slightly different model of implementing things (you’ll like the term “ephemeral”, which Amazon have chosen for a reason!).

So now the next question would be: How much does it cost. Is it really saving costs as many people state? Frankly, I don’t know yet. It may do. Surely, it reduces upfront costs, as Amazon won’t charge any setup or recurring fees, unless you opt-in for their “Reserved Instance” schemes, which are actually significantly cheaper in the long run. For example, a Micro instance (640 MB Ram, 1.7GHz Xeon; the smallest instance type) would usually be charged at US$ 0.025 per hour, which equals to US$ 219 per year if running full-time. If you commit to a year, paying US$ 54 one-off, your hourly rate is reduced to US$ 0.01, which together equals to yearly costs of US$ 117.60 or less than US$ 10 per month! That’s a smashing 50% discount almost. Higher discounts possible, if you can commit to 3 years.

However, the pricing is somewhat difficult to decipher and costs impossible to predict. I don’t actually know yet, how many IOPs (I/O operations) my EBS volumes and snapshots will generate. I can’t exactly tell how much S3 storage I will use. Also, I don’t know exactly what to expect on the inter-availability-zone traffic scale. Or the RDS (MySQL) read/write operations. Surely, previous monitoring gives me very rough estimates, but not good enough to make an educated guess as to what costs to expect on EC2. I will have to keep an eye on that over the next weeks and months, and also find some tools to get all these figures from the usage reports (CSV or XML files, downloadable from Amazon). At least you can see how your usage translates into actual costs for the current billing period, updated every few hours. So the costs wouldn’t hit you as a big surprise

On the plus side, you never pay for any over-capacity, which you would need to account for, if you built everything in-house. When you build infrastructure like that on your own, there are different things, which scale more or less dynamically (if you’ve got 10 servers already, buying two more doesn’t do a harm). But you’ve also got devices where upfront costs are enormous, because you buy them from the point of view, of what you might need in the foreseeable future, not what you do need at this very moment. Storage devices are a good example. A chassis from NetApp with only a few drives cost you an arm and a leg; then you can scale it a bit for a reasonable price; and then you’ll need another one sooner or later. But you always end up paying for more than you actually use at any given point. Same for networking devices.

Clouds like EC2 take those massive entry-costs from you (and your customers), which saves painful budget discussions. They’ve got a brilliant, scalable infrastructure, and one would be bold to assume that you could build anything better at a reasonable price (also take availability in distinct and independent data centres into account!). Now that they’ve got their PCI DSS certification, one of the biggest remaining concerns (what about data security in a proprietary, shared environment?) for many customers is gone, too.

I’ll go through the other proofs of concept on my list, see how usage translates into actual costs over time, and may then be able to add some very interesting offers to my company’s portfolio. Stay tuned

Surely, the cloud is not the solution to all problems (although it’s commonly propagated as that), but with decent automation and tools it can improve or at least add value to a variety of services.

Let’s say you are running FreeBSD on a server, and you need to do a major upgrade (that is from 6.x to 7.x). This process can take ages, if your machine is not running the latest hardware, and/or you have a lot of 3rd party software installed (ports). I’m not talking about an impatient person’s definition of ages, or about the one of a customer, who claims hundreds of quid financial loss in 20 minutes downtime on Sunday morning 1:30 am.  I’m talking about ages as in many hours.

Of course, a FreeBSD upgrade doesn’t require to be offline while it’s proceeding. But you will need to reboot. And as a rule of thumb, one can assume that dependencies in the ports will break. Usually only one or two of them, but it requires manual work, and can cause an unpredictable partial downtime, which is longer than it takes to reboot the machine.

So how can virtualisation help here? In a nutshell, it allows you to do the whole upgrade on another virtual machine. You can take a snapshot of the production machine, start it as a new VM, and do your work there, while the original VM stays online.

This also reduces stress enormously, because if you break something during the upgrade, there’s no time pressure to fix it. You can spend as much time as it takes to finish your work properly. Cool, isn’t it?

And when you’ve finished your work, you can inform your customer about an upcoming 1 or 2 minutes downtime for a major system upgrade (which you have already finished).

All you need to do when the time has come, is to sync files which changed during run-time (for example mail folders), change the network settings in order to make your upgraded snapshot take over, and then you can safely decommission the old VM. It really is as easy as that.