This article first appeared on mig5's blog on 27th April, 2010

The Aegir project has seen vigorous development leading up to, during and after DrupalCon San Francisco April 2010.

Unfortunately due to the volcano ash from Iceland, lead developer and Aegir author Adrian Rossouw got marooned in Frankfurt and was unable to join the rest of the team at Moscone.

When Aegir 0.3 was released and I joined the development team, Adrian mentioned in the release notes that the 'bus factor' threatening to affect the progress of the project was reduced thanks to have extra hands on deck. In a way, although it was a devastating disappointment that he was unable to make it to DrupalCon, the opportunity arose to test what we renamed to be 'the volcano factor' and I think we did well under the circumstances.

On the initial Code Sprint Sunday, Antoine and I sat down and ripped apart the Hostmaster install profile wizard. After a good 5 or 6 hours, Aegir became capable of being installed from the commandline directly, with no need to progress through the traditional browser-based wizard. Significantly, this install profile task wizard has always been one of the most common areas that new users struggle with when installing Aegir, as steps can be missed or misunderstood very easily.

Now with the new install script (which is currently compatible with our dev-services branch in git), one need only install the usual dependencies (such as Apache, MySQL etc) and then execute the install.sh script, which takes care of the rest. Seconds later, a Login URL is given to you and you can immediately access the Aegir frontend!

Amongst the sessions on Monday, we also participated in an informal Aegir BOF session that was organised by Brian Wood, much to our delight. In this session, we gave a brief overview on where Aegir was headed, but focused on hearing from you guys, our user base, on how you're using Aegir and what issues / benefits you've experienced that we could take back with us.

That evening after the BOF, we organised an informal Aegir dinner to carry on further discussions. I had a very nice vegetarian Pad Thai, and later we made our way to the Lullabot and official DrupalCon parties :)

As we approached Wednesday, the Aegir team knuckled down and actually got around to preparing an agenda for our session that got voted in (thanks!). For myself this also meant preparing a local dev demo environment to demonstrate the latest features in Aegir, as well as testing all the demonstrations we'd exhibit and make sure they actually worked :)

Somewhere along the line, both Antoine and myself were interviewed by Lullabot for their famous Drupal Voices podcast series, where we discussed Aegir, best practices for dev/stage/live workflows, Drush and Drush Make. Expect to hear these interviews in the coming months as they trickle into the Drupalsphere.

For good measure, Adrian discovered why our once-working Drupal7 site provisioning was broken (as we expected, it was due to changes upstream in d7 core) and fixed the issue. At that point we released 0.4 alpha7, just in time prior to our session :)

On Wednesday 21st April at 16:15pm, we presented the "Aegir - One Drupal to Rule Them All" session, covering:

1) a basic introduction into the project
2) features/functions of aegir
3) what's changed since Paris
4) development/security practices
5) what's coming
6) some Q/A time

We felt the session was a success and it was great to see so many people show up and ask a lot of good questions.

You can watch the full session here or download/watch on archive.org

After the conference, we made it to the Thursday code sprint at San Francisco State University, and nailed a few more bugs / features! What was also discovered was that one can upgrade a vanilla Drupal 6.16 site to the latest Drupal 7 using our Migrate feature in Aegir, now that an upgrade path exists and drush updatedb succeeds. Wow!

DrupalCon SF 2010 is over but progress continues. All the bulk of our hard work at the moment is within the "dev-services" branches of the provision and hostmaster git repositories on git.aegirproject.org, where our new object-oriented refactoring and introduction of the abstracted 'server' concept and service-based API is paving the way for true multi-server support.. not far away now!

You're welcome to checkout these branches and start playing with it and fix some issues, or even just to test out the new command-line-only installation process. However, things are changing all the time, so it's volatile and there may be things that are very broken. Definitely don't use the dev-services branches in production!

Thanks to all who made it to DrupalCon or watched from a distance, attended or are listening/watching our session, came and introduced themselves and/or had a beer with us at some point, or are simply just using Aegir and loving it :)

I've some more exciting things to discuss in Aegirworld, but those will have to wait :)

Hope to see you at the next DrupalCon!

Edit: I forgot to also give thanks to two other groups:

1) the organising committee for DrupalCon itself - awesome job!

2) and also the guys at Koumbit for sending along a great team including Antoine, and equipping them with great Aegir 'One Drupal to Rule them All' stickers :)

This article first appeared on mig5's blog on 13th October, 2009

Tonight at 18:30pm GMT+11 I'll be giving a talk/presentation on 'An Introduction to the Aegir Hosting system' at the regular monthly Melbourne Drupal meet @ the Emspace offices.

My notes during the last week kind of took on a life of their own, and so in anticipation of the talk, I thought I'd publish them here, despite the somewhat casual tone, for those who can't make it and want a somewhat (perhaps overly, for an Introduction!) detailed overview of the Aegir system and what it does.

I believe the session will also be recorded, so we should see the video pop up on blip.tv or somewhere in days to come. Update 20/10/09: The video is available here http://blip.tv/file/2741212

A variation of these notes may end up being part of the documentation on g.d.o too.

Update: Here's a PDF of these notes http://ln-s.net/4Nbk

---------------------------------------------------------------------------

An introduction to the Aegir Hosting system

Contents

1. What is Ægir?
2. Brief history of Ægir
3. Dissecting gods: the anatomy of Ægir
   
   • Frontend
   • Backend
   • How Hosting and Provision work together
4. Features and terminology
   • Everything is a node
   • Platforms & Sites
   • Task queue
   • Drush, hooks, rollback
   • Installing sites
   • Site Aliases
   • Importing sites
   • Verify, Packages (schemas)
   • Site Disable/Enable
   • Delete
   • Migrate
   • Backup
   • Restore
   • Deploy
   • Clone
   • Batch Migrate
   • Cron Queue
   • Clients, roles, permissions
   • Ports & SSL
5. What Aegir doesn't do
6. Roadmap - where are we going
7. Questions
8. Links

What is Ægir?

Ægir is a set of Drupal components that when used together, help you manage other Drupal sites.
It does this by providing you with a simple Drupal based hosting frontend or control panel that allows you to perform actions against your network of Drupal sites. Some of these actions or 'tasks' include Installing new sites, Verifying the integrity of sites, backing up and restore, management of 'Platforms' (which are copies of Core or distributions such as OpenAtrium), and migration of your sites between these platforms. We'll cover the relationship between sites and platforms in a moment.

To Ægir, a site is just a node like any other node in a Drupal site. Depending on the node type, be it Site or Platform etc, determines what tasks like these can be performed.

Taken from the page on groups.drupal.org: In Norse mythology, Ægir was the god of the oceans and if Drupal is a drop of water, Ægir is the deity of large bodies of water.

Brief history of Ægir

Ægir used to be called Hostmaster, or Hostmaster2, and was authored by Adrian Rossouw who remains the lead developer of the project, back when he worked at Bryght. I believe Bryght pioneered the idea of a system like this to manage large deployments of sites for their clients, and Hostmaster was the result of that work and was subject to much growth and enhancements over a period of 4 years before it ever became 'Aegir' as it is now. Adrian's also known for other Drupal accomplishments such as the Forms API, the PHPTemplate theme engine, and the install profile system.

Since then, Bryght got acquired by Raincity Studios, and Adrian's now working for Development Seed.. thus a large influence on the development/direction of Ægir is naturally coming from that camp as they provide Adrian with the time and the means to work on improving the product.

Joining Ægir as part of the core development team is Antoine Beaupré, aka 'anarcat', who's from the Koumbit worker's collective in Montreal and been working on this for a long time too, and myself.

For this introductory talk, I'm going to focus on the anatomy of Ægir and how the components fit together and with Drupal. Then I'll provide a brief overview of the main features of Ægir and what you'd use it for.

After the talk, to avoid taking up too much time, I'll be happy to run through some more in depth, under-the-hood areas of Ægir such as the Provision backend, the database schema etc, and answer any of the more technical questions you might have.

Dissecting gods: the anatomy of Ægir

Ægir is not a 'module' nor yet a 'distribution' for Drupal, but currently a collection of 'components' as I like to call them, that work together to do the whole job.

In order to best understand how it all works, we can define Ægir as made up of two parts: the frontend, and the backend.

Frontend

The frontend is actually a Drupal site that makes use of a module called 'Hosting', which does all the job of the frontend: basically it provides the mechanisms to add / remove nodes like any other Drupal site, allowing you to point and click on tasks like Install, Backup and so forth. It also provides a nice UI to manage your sites in general, and to review tasks that have completed, or in progress, or failed for whatever reason. Most screenshots you've probably seen of Ægir, are using the Eldir theme developed by Young Hahn at Development Seed, which is made specifically for Ægir, though it is optional and Ægir will function without it.

Backend

The backend is made up of a 'drush module' or extension called Provision, which does pretty much its namesake: it works with Drush very heavily to handle all of the tasks such as installing new drupal sites, importing existing ones, installing platforms, verifying everything, instigating backups and restores, and so on. So it's actually doing the heavy lifting here, actually executing the tasks you assign to the system.

Provision is invoked through the use of drush commands, and thus these commands can also generally be executed from the command line to perform the functions that make Ægir so important. In fact it's very much by design that all the magic happens in the backend, because it allows us to have one Ægir frontend that can manage multiple backends, or in otherwords, multiple webservers and database servers. Though these are experimental features at the moment.

How Hosting and Provision work together

The glue that binds what Hosting lets you kick off tasks for, and how Provision learns of those tasks and executes them, is by way of adding the tasks to a Task queue in the database, which is viewable in the frontend.

On your server, we then have a dedicated 'aegir' user, whose crontab has an entry to regularly look and 'dispatch' any queues containing tasks that are pending, which are then forked off into drush provision commands in the background.

So the idea is that all this happens behind the scenes for you as the manager of the aegir system: the frontend exists for you to queue up all these sorts of actions, and let Ægir worry about actually doing them in the backend.

Along with Provision, Hosting, and Eldir, there is also another component which is an Install Profile called 'Hostmaster'. The Hostmaster Install profile is, like all install profiles, really only executed once, when you are setting up Ægir for the first time. And it does the work of getting information from you, or telling you to set things up on a system level (like directories, permissions etc), asking what Features you'd like to enable, and kicking off the initial verification of the system, setting up the crontab entry etc.

The home directory of Aegir

The backend Provision module

The Hostmaster profile and the Hosting frontend module contained therein

The Features and terminology of Ægir

So before I start running a few examples of what Ægir can do, I thought I'd bore you some more with the terminology and features that Ægir has, as there's often some confusion over what some of the terms mean.

Some features in Ægir are enabled by default, some cannot be turned off, some are optional and can be enabled at any time, and some are experimental and may have unexpected results in production environments.

Everything is a node

In keeping with strong Drupal tradition, and out of pure sense, everything is a node in Ægir. This includes Ægir's information on your web server, database server, platforms, sites, right down to tasks, and information about modules, themes, and install profiles.

Platforms and Sites

First of all, Ægir's very much designed to work within Drupal's multisite structure. Often one of the biggest hurdles users experience when trying to learn Ægir, can be traced back to the fact that few of them are actually making use of Drupal's multisite design, and instead tend to use one core per site. While Ægir will work that way just fine, its designed to make use of the proper multisite logic introduced way back in I think Drupal 4.5 or maybe d5 - in fact, it's what makes it so powerful and useful - it handles multisite for you, and it handles it properly.

For whatever reason, multisite is often not used, and so users are often confused about what constitutes a 'Platform' in Ægir, so we'll cover that now.

Basically the first thing you do after installing Ægir, is go and set up what we call a 'Platform', which is the code base within which you'll create a Site to sit on.

'Platform' translates directly to a copy of Drupal core, like Drupal 6.14 or 5.20, but it isn't limited to Drupal core. It could very well be a 'distribution' or custom Drupal core like OpenAtrium, Pressflow, OpenPublish, and so on. So it's important to recognise that a Platform is actually very simple to understand conceptually once you learn to exchange the term for a 'Drupal core' or thereabouts, and I suspect it's often expected to be more complicated than it really is.

So just per a standard multisite system in Drupal, sites are installed into the sites/ subdirectory of that Platform.

So a Platform and a Site are two examples of what are simply node types in the database, just like any regular Drupal site. This is what makes Ægir so easy, because to create a new Site, you can literally node/add/site or Create Content > Site and fill out the required fields, submit the form, and Ægir does the rest.

So you start to form a picture in your mind of a pyramid structure: In Drupal multisite, a Core or a Platform manages many sites. Ægir is a level above that, in that it can manage many Platforms. So Ægir becomes very powerful, because through inheritance of managing many Cores or Platforms, it exponentially can manage many sites, be they Drupal 5, Drupal 6, Pressflow, or Openatrium sites. And so on.

The task queue

When you add a new Platform or site, you can see that these items are immediately dropped into the task queue which you can see in this block in the sidebar. As the cron comes around once per minute, it'll pick these tasks up and dispatch them to the backend to be executed.

Depending on the return from Drush and Provision as they execute these tasks, the status of this task in the queue will change from Queued to either Failed or Success. In both cases, the full drush log output is given in the task node.

On a failed task, you can review the problems and if you feel you've fixed the issue, re-schedule the task to re-enter the queue and be executed again.

Drush and rolling back changes

Because Drush has a rich API that provides validate_, pre_, post_ and rollback hooks, we can safely revert any changes that were made if a task fails, through a rollback, or perform other functionality before or after the fact. This means that although Ægir is taking a lot of the heavy lifting out of your hands with this stuff, it's also very careful with your data. It doesn't want to destroy your sites, it wants to nurture them, and it'll undo or refuse to perform any actions it thinks might be unsafe.

Installing Sites

In the case of installing a site, support is written for Drupal 5 and Drupal 6 platforms. There is Drupal 7 support in there that once worked, but the nature of Drupal 7's shifting meant that support for d7 fell out of Drush, and thus it currently doesn't work in Ægir, though I think a Drupal 7 platform *will* verify ok.

Installing sites works because Provision basically knows how to create a mySQL database and credentials automatically for the site and apply these settings, along with locale, timezone etc, and any other settings you normally enter manually during a Drupal install, automatically into the installer. It doesn't matter what installer it is: it can be either the standard Drupal default installer, or a custom install profile you've written, or other install profiles such as the atrium_installer profile that makes OpenAtrium what it is.

Install also goes through and generates the Apache vhost configuration file for the site and restarts Apache, so you don't need to do any of this.

On successful installation of a site, the onetime login reset link will be e-mailed to the 'Client' (we'll go through Clients later), and the login url will also be logged to the task output for quick reference. It's normally the task output that I'll go to to retrieve the login url. So immediately you can login to your brand new drupal site, and you only clicked a couple of buttons.

Site Aliases

Ægir supports site 'aliases', which translate directly to the ServerAlias parameter in Apache vhost files. One can add URL aliases that will serve a site called by another name. A classic case is installing a site with a URL of 'www.example.com' with an alias of 'example.com' so that example.com also serves the site (providing a RR is set in DNS of course).

Redirection is also supported as an optional feature to Aliases, allowing permanent redirects of the aliases to the main site URL by way of mod_rewrite. All this is configurable in the site node form, though Site Aliasing is an optional feature that must be turned on first.

Importing sites

It is possible to import existing sites that you're hosting on a server, into your new Ægir system.

Since most people seem to be using one core per site and not using multisite, it's just a matter of moving the copy of core into /var/aegir/ , renaming sites/greenbeedigital.com.au to sites/$yoursite, and then adding a Platform in the Ægir frontend. Upon a successful Verify of the new Platform, any detected sites living in the sites/ folder will have an Import task spawned for them, and Ægir will hence 'learn' of these sites and their packages.

Obviously it is not ideal to have one platform per site, and the whole point of Ægir is to support multiple sites on one Platform. So once this site is imported, you will likely want to Migrate the site to another platform, which we'll cover in a moment.

Verify, and Packages
Along with Install, other tasks exist in Ægir. One that is often run is Verify, which just checks that the filesystem of the site or platform looks sane, permissions all correctly set, confirms the aegir superuser can create new databases, and so on. it also refreshes the database with knowledge of what modules, themes or profiles (altogether referred to as Packages in Ægir) exist on a platform of site. It keeps a running state of what packages are installed, so that if you try to Migrate a site, it knows whether the target platform can support these modules or not.

If you edit a site or platform's node - for instance if you are adding a URL alias for an existing site - on node_save, the item will be re-added to the task queue as a Verify task. Such functions serve to handle such cases like an alias addition, where the site's vhost needs to be edited and Apache restarted for the change to take effect.

Site Disable / Enable

If a site is installed and enabled, one can Disable the site, which effectively puts a redirect on the apache vhost for that site to redirect to a page saying the site's been disabled. It also goes and backs up the site.

If a site is Disabled, it can then also be re-enabled, which essentially unsets the redirection in the vhost so that the site starts working again.

Site Delete

After a site is in a Disabled state, you can Delete a site, which will go through and (again) take a backup first, then goes and removes the actual site data from the server, removes the knowledge of the site from the aegir database, drops the site's own database, and removes the vhost config. As mentioned earlier, if any of these steps go downhill, the procedure will be reversed as best as it can.

Site Migrate

I mentioned Migrate earlier, and this is really in my opinion, the biggest feature of Ægir and what makes it so worth using. Migrate essentially is another task that tells Provision to *move* the site from one platform to another. The most obvious case of where you'd want to do this is when a new Drupal core release comes out, like recently with 6.14.

All I had to do is download Drupal 6.14 to the server, add it as a platform in Ægir, and then use Migrate to move my Drupal 6.13 sites to Drupal 6.14.

In the process of this, it runs the Drupal updates and hence performs the hook_updates, schema changes etc that one would normally do by hand in an upgrade.

What makes quite significant is that it's by design that you can take a Drupal 5 site, and use Migrate to upgrade it to Drupal 6. It's built right into Ægir to handle that upgrade automatically for you.

Migrate basically takes a backup of the site, and then it uses a special 'deploy' command to 'deploy' this instance of the site to the target platform. It then removes the site from the existing platform.

I mentioned earlier the concept of a 'Package' in Ægir, which is any module or theme or install profile that Ægir finds on a platform or a site. With its knowledge of the package, its schema version and what 'instance' in which this package is being used, it can analyse a site that you want to migrate, compare the packages in use with what can be used on the target platform, and present a report on whether the site can be migrated or not.

In other words, it will intelligently calculate that you cannot migrate a Drupal 6 site to a Drupal 5 site, because the schema versions of Drupal 5 packages are incompatible with those installed on the Drupal 6 site. However obviously, the reverse ought to work.

And yes, it backs up and will rollback if unsuccessful. :)

Site Backup

So I've mentioned Backup a few times, so I should cover it: essentially the Backup task can be executed against a Site. It does a mysqldump of the site into a file in the site's directory, and then it tars up the whole site's directory and drops the tarball into a backups directory that lives outside the document root of any platform with a timestamp.

Site Restore

Using the Restore task, you can take any of these backups, and with a single click, restore your entire site, codebase and database all together, to a previous state from one of the tarballs.

Deploy

The 'Deploy' task I mentioned earlier isn't shown in the frontend. It's a backend command that is used in Migrate and Clone tasks. Clone is a new feature in our early 0.4 alpha releases that does what you probably are already thinking it does: it generates a snapshot of a site (actually using the Backup task above, we reuse as much code as possible) and uses Deploy to deploy that copy of a site with a new URL to any platform, be it the current platform or a separate platform.

Site Clone

So you can use Clone to clone entire copies of your site to a new URL, which could be useful for testing, or if you have a sort of 'template' site with common modules and themes, structures or common nodes etc. You could consider it a poor man's install profile.

Batch Migrate

To date, you could only Migrate a site on a per-site basis. We've recently introduced Batch migrate, which means you can migrate *every* site that currently exists on a platform, and move them all to a new platform in one hit. Again, especially useful when you have a lot of sites on a Drupal core that has suddenly become vulnerable with a new security release, and you need to move them all across to the new core as soon as you can.

We know of Ægir being used to deploy and manage more than 2100 sites on one system, actually by another Australian, Dave Hall, who's up in Bendigo, and goes by the alias skwashd on Drupal.org and IRC. So features like Batch Migrate are especially significant.

Cron Queue

Since Ægir already has a task queue scheduler, it makes sense that it should be able to queue up enabled sites and run cron regularly on the sites themselves. So a Feature exists to do exactly that. Both the regularity of the Cron and the Task queues are configurable.

Clients, roles, permissions

Another Feature and node type is that of the Client. A Client in Ægir is not a specific user in the typical Drupal sense, but more so a conceptual wrapper node that also translates directly to a role type. One then can create users and assign them to a 'Client' just like a role. Such users have the ability to login to the Ægir site and manage their own sites. In this way you can expose Aegir functionality to your clients and let them create their own sites or manage them (to a point).

Another role that Ægir sets up is the Ægir Account Manager: this is design as a non-technical role, likely a member of your own staff, who has the ability to set up and manage Clients in Ægir, but cannot perform any technical tasks or install sites etc.

Ports and SSL

A lot of work has been done in recent releases to implement multiple webserver Port support, along with SSL support which is not quite complete.

Currently one can create only one site URL on one port, but cannot create the same site URL on another port. There are cases where this will be required (such as port 80 and 443 for sites that need SSL as well), which is something I'm working at the moment in my development.

What Aegir doesn't do

• Aegir knows about your modules and their release schema etc, in order to handle migrations and that sort of thing. But it isn't a plugin manager, and you can't disable/enable modules of sites within Ægir.
• To the dismay of at least 50% of users who install it, Ægir is not in fact a build management tool, it's a site management tool. But one can build Platforms using drush_make, which can pull from drupal.org repos as well as your own git/svn/cvs repository, and migrate your sites onto new 'release' builds. I'll spend some time on this after the talk for those who are interested.
• It doesn't provision non-Drupal sites. But it may support static sites, Joomla? in the future..

Roadmap - where are we going?

• Lots of UI refactoring will be done before 1.0 - modal dialog, more UI feedback to the user
• Refactoring 'Servers' from being separate node types i.e webserver, dbserver etc, to a single 'Server' type that acts as a container for pluggable Services (http, mysql, postgresql, dns)
• Develop a File service, one reason is for dealing with moving backups around.. 'spoke' model or 'mesh' model (see roadmap)
• Just in this week gone, in the alpha2 release, we leverage 'Drush make', allowing Hostmaster to provision itself and migrate itself to new platforms (0.4 alpha2 and onwards likely)
• Third-party application hook-ins, i.e DNS, LDAP, Mail, Jabber.. with the idea of a control panel done right
• Stronger quota management and the potential for e-commerce hook-ins, selling sites as a product
• Move to PDO to support PostgreSQL, SQLite etc
• The 1.0 release will see a 'frozen' Hostmaster API

What else?

Questions?

Links

• http://groups.drupal.org/aegir
• http://drupal.org/project/hostmaster
• http://drupal.org/project/hosting
• http://drupal.org/project/provision
• http://drupal.org/project/eldir
• http://drupal.org/project/drush
• http://drupal.org/project/drush_make
• Aegir issue queues
• Migrate your Drupal sites to a new Drupal core security release with Ægir
• Automate your site maintenance troubles away with the Aegir hosting system - presentation by Adrian and Antoine at DrupalCon Paris
• 0.4 Roadmap

This article first appeared on mig5's blog on 28th October, 2009.

It's the million dollar question. And it's rarely been answered because it's so darn hard to do so.

Today I'd like to tell you how I do it with my favourite deployment weapons: Aegir, Version control, and Drush Make.


This'll be a rather low-level article designed for power-users. For a more high-level overview of the benefits described, check out the recent blog post by Development Seed.

The Problem

While many or most of us are using version control to keep track of our code and so on, a problem still persists, and it is twofold:

1) A lot of important data gets stored in the database in many database-driven content management systems, such as Drupal
2) The transition of updates through version control to the site can be a royal PITA.

Recently 1) has pretty much been conquered thanks to Features, which provides the ability to export the 'mechanics' of a Drupal site from the database into module-like code, which can then be version-controlled.

Adrian Rossouw covered how Features can help solve the classic Dev / Stage / Production workflow problem in a great blog post. Features is one of those big guns we've all been waiting for. I mean, google 'development live workflow', and it's this post that is number 1.

With the rising popularity of the Aegir hosting system, and its ability to fire off sites like a proverbial machine gun, the other question is becoming even more important.

Because Aegir is this fantastic system for managing your sites, right? Of course, I would say that. But how can you keep a site under version control when Aegir's capable of making and manipulating sites almost on-the-fly with the click of a button?

The Solution

Enter: Drush Make

The answer has come in the form of Drush Make, which provides an extension to Drush, the command line tool I often refer to as 'apt-get' for Drupal.

What Drush Make does, is through the implementation of flat files (similar in appearance to .info files from modules), provide a mechanism to fetch data from practically any source you tell it to, and do $stuff.

This can include downloading Drupal core, as well as a bunch of contrib modules from drupal.org . However, it also supports the ability to fetch:

• code from CVS repos, including tag-specific
• ditto for svn
• ditto for git, including checking out a particular branch after cloning the repo
• grabs .tar.gz stuff, wget-style, useful for libraries
• can fetch and apply patches

Sure, you say, that's great. It's basically doing batch drush dl's with some extra features/protocols thrown in. But how does this help with version control? Not only that, but how is that going to help keep this stuff under control while managing the site in Aegir?

Certainly for me, much of the issue has been that Aegir is inherently 'on the fly' with its actions. It has no concept of version control. But eventually I realised I had to stop expecting Aegir to handle this aspect, and realise that I could work with it by bringing version-controlled data to it for processing.

Or, put another way, I realised the very act of moving data around, does not need to be a version-control procedure in itself.

One must start to think like Aegir: and that is that the Platform and its application is the important data, that needs to be controlled. The site is just an instance of the application, and so long as your application is controlled, your site can be manipulated as an instance of it.

Adrian mentioned it in his podcast interview with Lullabot on October 15th:

It helps to think of install profiles as applications, and sites as instances of those applications.

If you can grasp that, you're on the home stretch already.

The Platform becomes the Platform Build.

One of the concepts I struggle with the most when training users on Aegir, is that of the 'Platform'. It turns out that many developers haven't been using Drupal's multisite design even prior to Aegir, despite the fact that the design's been around forever. So the idea of having multiple sites inside the /sites/ folder of Drupal core - in other words, sharing the core codebase between sites - is somehow (often, but not always) already enough of a paradigm shift.

In Aegir, one creates Platforms, which are just code bases, or a copy of Drupal core (or Open Atrium, etc). Only then can we create and manage Sites, which live on Platforms.

The analogy is directly like this:

/var/aegir/drupal-6.14 << this the core, the Platform
/var/aegir/drupal-6.14/sites/www.mig5.net << this is the site, 'on' or 'inside' the Platform

What makes this hard is that some users are still getting used to this idea, even though Aegir isn't doing anything different here. It's just exposing that multisite design of Drupal, by making it easier.

If you can grasp the idea of multisite, you'll find it easier to understand how Aegir works with your sites, and how a workflow can be developed to handle build management.

I mentioned Drush Make, and it's ability to grab the Drupal core.
And I mentioned shifting one's mindset to think of Install profiles as just 'Profiles', or 'Applications', and sites as instances of those Applications.

With this in mind, a bigger picture starts to form regarding the potential of Drush Make, since it can, in one fell swoop,

• download the core,
• fetch the install profile from your git repo
• fetch some contrib modules
• fetch some custom themes and modules from your other repos
• maybe apply a patch against a module
• fetch some extra third-party libraries, i.e jquery.ui, or geshi

No, really. Look at this (entry-level) example of mig5_net.build:

Executable from the command line like this:

My install profile or 'application' is the git repo that I'm cloning. Note the branch reference: after Drush_make 'git clone's it, it'll check out a specific branch, or 'build'. This is how this release is different from past and future builds: it's a specific release I'm building for.

Suddenly with this one execution of a single flat file, you've built a copy of Drupal core, and dropped in the profile or 'application' into its /profiles/ directory. That's a platform in Aegir's eyes, and not just any platform: but what I call a 'Platform build'. It's a specific release of a your application, right from the top of core, down to the last css file of your site's theme.

Hang on, .css file? Where's the contrib and all that? You only cloned the install profile from git

Yes, in the makefile above, I'm only grabbing the core and my install profile application. Here's the secret: it turns out Drush Make has a recursive nature! I tuck away another makefile inside my install profile repo, called mig5_net.make, and this is the real gem.

What we have here is a series of contrib modules from drupal.org. Also I have a slightly hacked version of singular (css stuff), so I don't use Development Seed's feature server repo to grab this: I keep my own copy in my own git server. However, I *do* use the Tao base theme from their feature server, because I know, like contrib, I don't make any changes to it.

Alternatively, if I was smarter, I could have grabbed Singular from DevSeed's feature server, and written a patch for the css change, and asked Drush Make to apply the patch instead. Here's a snippet of such a thing, taken from the Managing News makefile in its install profile, where the Context module is patched:

...just so you don't think I'm making this stuff up :) Yes, such features exist, and are in active use!

So, this make file gets downloaded when the install profile git repo gets cloned and the build-specific branch is checked out. While Drush Make grabs the profile from the first makefile (I call it the .build file), it scans the directory and finds this make file, and recursively executes it - thus grabbing all the site-specific components in the process.

Time to tell Aegir about the new Platform build

Suddenly what I have is the entire platform, complete with any updated components I made in the makefile of that specific branch in git. At this point, I just have to tell Aegir about it!

So I go to Create Content > Platform in Aegir, give it a name (usually suffixed with the build number I give it as above), and tell it the path to the new platform I made (/var/aegir/drupal-6.14_build_2009101601)

Aegir goes and checks the platform, learns about it, adds the package information it finds within it to its database, and if all's well, gives me the big green tick.

But where's the settings.php? The site files? This isn't actually the *site*!

Exactly. It's only the application that we've built, and it's only the application we store in version control and generate with makefiles. The site still exists on the old platform, and Aegir knows it does. What we have to do here is stop thinking of the /sites/www.mig5.net/settings.php etc as the *site*, but rather, the current running 'instance' of the application.

It sounds a bit weird, doesn't it? But you don't realise, if you're already using Aegir, you're already probably doing this already. Drupal 6.14 came out recently, and your sites were on 6.13. You built a new platform (6.14, by downloading it), told Aegir about it, and you used the Migrate task to upgrade all your 6.13 sites to 6.14 at the click of a button.

We're doing exactly the same thing here: you've got your new platform, with your updates. Now you just need to Migrate the site onto that new build. Your settings.php and files from /sites/www.mig5.net/ get carried across to the new build, and any component code/database updates will get applied in the process.

Suddenly, you've just exercised the perfect workflow:

• You kept the application, important stuff, in version control
• You managed to keep *less* of a) in version control because drush make can fetch the contrib you don't hack, from drupal.org and other sources
• You built the whole new release with just one command on the shell
• You managed to seamlessly upgrade your site onto the new release, applying all the updates, and
• you did that last one via Aegir, which means you were protected with the rollback functionality of Provision and Drush, meaning if it all went pear-shaped, you'd be back on the previous release as though nothing had happened.

This is how Aegir can cooperate with version control: all it needs from you is to get that Platform, or build, in place, so it can use it and take care of the release process for you.

And while it used to be difficult to build an entire distribution complete with all the nitty gritty components required, it's now never been easier to do this step in a matter of seconds, because Drush Make does this job for you.

This application, www.mig5.net, is literally two files in my git repository, plus the theme (I hacked on Young Hahn's Singular theme a bit). The two files are the install profile itself (which, of course, I only ever once: first time install of the site) and the mig5_net.make file, that does everything else.

Dealing with the dev > live process

This really packs a powerful punch when you're working on a development version of your site to make changes prior to sending live (and if you're not, you should be). The process above fits perfectly into this model. This is how one does it:

Need to make changes?

Dev

• Make a new build using drush_make to grab the latest bits of everything. Add that platform to Aegir. I might call this build 'drupal-6.14_build_dev_2009102601'
• Use the 'Clone' feature to clone your current Live site to the test build.
• Make your changes to the dev site (maybe you are changing the theme, or adding a module)
• Commit those changes back to the repo (the theme changes, maybe adding that new module as a ''project' in your drush_make file inside your install profile). Make a branch of the repo in preparation for a new live build.

Live

• Make a new build using drush_make to grab the latest bits of everything (this grabs the changes you just committed). Add that platform to Aegir. I might call this build 'drupal-6.14_build_live_2009102601'
• Use the 'Migrate' feature to migrate your live site to the new build, which picks up and applies any changes/upgrades

I shared some ideas with Adrian Simmons (@adrinux) who has taken the build/deployment logic I describe here and generated an absolutely fantastic workflow diagram showing the transition of data from dev to live, as well as the relationship between drush_make, version control, and Aegir. Here is the diagram - it does a hell of a better job explaining it all than me and my wordy ways :)

The status of Drush Make

Awesomeness.

No, you'd know by now that I'd say that, but that's actually the release notes for the 6.x-2.0-beta1 release, which Dmitri unleashed today, after an amazing night (well, my time) by Adrian Rossouw who obliterated a bunch of bugs after being given commit access. The Aegir project now depends on drush_make to build the frontend system and its relevant components, as of HEAD and the upcoming release 0.4-alpha3 (stay tuned for that very shortly). For this reason, we have a direct interest in the project and its future, and so we were lucky to work with Dmitri and squash a few issues very quickly - some of which were actually identified during the writing of this article :)

The recent changes to Drush Make include bringing it in line with changes in the also-recently-released Drush 2.1, adding support for checking out git branches after a git clone, and supporting absolute paths for destination of builds on the server.

Need to play with it some more to try it out? You can see examples of my 'build' makefiles in my git repo as well as my application install profile for mig5.net, which contains the 'bigger' drush_make file. Or check out the Managing News install profile and its make file - it's the first project other than Aegir to use a make file to build itself.

If you're new to Aegir, Drush and Drush make, I hope this article gives you the incentive to try it out and see why it rocks. If you're using Aegir and Drush already, I hope Drush Make makes more sense now, and that this gives you that 'ah hah!' moment if you've been scratching your head about how to handle build management in an Aegir environment.

(thanks to Adrian Simmons, Adrian Rossouw and Eric Gundersen for their contributions/ideas for this article, and most of all to Dmitri Gaskin for his mad skills.. thanks for changing the game with Drush Make!)

https://www.digitalocean.com/community/tutorials/how-to-install-aegir-an...

This article first appeared on mig5's blog on 9th June, 2011.

A while ago I wrote that I had started to use Jenkins for various purposes such as server backups, providing better notification in event of failure, yadda yadda.

Over at the Aegir project I implemented Jenkins to give us a continuous integration platform that kicks into gear either on demand or whenever one of us pushes a commit to the Drupal git repos. It does the following:

• Provisions a new virtual server (at Rackspace Cloud) via the Libcloud API
• Installs Aegir on the server
• Builds a Drupal 6, Drupal 7 and OpenAtrium platform via Drush Make
• Installs a site on each respective platform

A number of other build jobs exist, such as testing upgrades of earlier Aegir releases to latest versions, but I won't go into that here.

What several people have asked me to elaborate on, is that the combination of Jenkins, Cloud-based virtualisation and Aegir can make a very useful testing tool if you're building some sort of Drupal solution like a distribution or some sort of turnkey Drupal appliance where you want to repeatedly test the installation process (especially with a custom install profile, for example). This article will describe how we built such a thing.

Disclaimers before we start

1. I'll assume you have a server set up with Jenkins installed already, as that's out of the scope of this article.

2. The code I'm giving you uses Libcloud API, but it is very much designed for the Rackspace Cloud driver implementation. I know for a fact, through building other tools, that other provider drivers such as Linode have differing requirements (Linode, for example, requires you to also specify a datacentre to build in). Please don't contact me complaining that it didn't work with Provider X, because I'm just showing you how we use it with Rackspace. Since there's a good chance you're smarter than me, you're welcome to take the code and abstract it properly :)

3. The code is far from perfect because I'm rather crap at such things. The error handling is extremely crude.

4. I am not going to explain every line of code in this article: this is targeted towards clever people who will read the code and understand what to tweak, what assumptions to add or remove to make it build their own project. Again, this is just what (currently) works for us.

5. I am assuming the Jenkins server is running on Debian Squeeze, where things like fabric and python-setuptools can simply be apt-get installed with no fuss.

Step 1) Install libcloud tools

Libcloud will provide the API necessary for provisioning a brand new virtual server at a provider such as Rackspace Cloud.

Run these commands as root / with sudo on your Jenkins server.

You'll need python-setuptools on a Debian/Ubuntu system:

Install libcloud likeso:

Step 2) Install fabric

Fabric, a python library for communicating with servers over SSH, will run necessary commands to the new VPS to install Aegir and the rest, via SSH.

Run this command as root / with sudo on your Jenkins server.

Step 3) Upload scripts to the Jenkins server and edit

Two files provide the magic that does the whole job of building a new VPS and installing Aegir and the rest. You want these two files on the Jenkins server for the Jenkins user to use.

The .ini file is for you to edit and enter your API keys, server specs and relevant details such as your email address, trusted IP for firewall whitelisting on the new VPS, and URL to fetch your makefiles from to build your project.

The other file with the .py extension is the actual python code that Jenkins will execute to build the whole thing.

You can clone my repo containing the two files from github:

The code assumes this directory containing those files are in the home directory of the Jenkins user, e.g /var/lib/jenkins/aegir_ci

It's at this point that you want to edit the aegir_6.x-1.1_install.py file and define what to build. In my example that I'm giving you, I'm fetching three Drush make stub or 'build' files from my github repo, named drupal6, drupal7 and openatrium .build respectively.

The code that does this is the function fab_install_platform().

Thus, this code is very much specific to building your project from a makefile, the fact that the makefile is fetchable over HTTP (per the ini config file, it fetches from my github repo) and also makes some assumptions on what install profile to use when installing the site.

The code that installs the site is the function fab_install_site(), and takes two arguments: the platform per the above (e.g drupal6, drupal7 or openatrium), and the install profile name (default, standard or openatrium in this example). So this is where you want to change the install profile name and only install whatever platforms you want, based on the name of your makefile.

Step 4) Add a new Jenkins job

Add a New Job of type 'free-style software project'.

In the combobox 'Add Build step', choose 'Execute shell'.

In the text field that appears, enter this line:

Choose any other settings you'd like in the job, such as notifications if a build fails, whether you want to automate this build on a scheduled basis (like a cron), and so on.

Step 5) Execute a Build

After you've saved the job, simply click 'Build Now' in the left sidebar and watch the Console Output to see the new server get provisioned, your Aegir installed and your distro or custom Drupal app built and a test site install processed.

Blowing it all away

My example code has a snippet at the bottom that completely destroys the VPS at the end. This is because our Aegir project builds on a scheduled basis, and that includes times where I'm asleep: I don't want these cruft VPS lingering around costing me money.

You may wish to prevent the auto-destruction so that you can login and take a look around, especially where there are issues. To do so, just comment out conn.destroy_node(node) at the bottom of the script.

To login to Aegir, either find the one-time login URL in the Console Output of the job itself, or check your e-mail, where you should have received the link in the welcome message.

This article first appeared on mig5's blog on 26th May, 2011

Today, Drupal released a security/bugfix for Drupal 6 and 7, bringing the latest version of Drupal 6 to 6.22.

Aegir is well known for making site upgrades very easy, fast and safe by way of the Migrate task. But some users might be wondering how to make the Aegir 'frontend', which is a Drupal-6 site itself, run on the latest version of Drupal without having to reinstall Aegir entirely.

For instance, if you installed Aegir 1.0 or 1.1, your Aegir system is probably running on Drupal 6.20.

Fortunately, our Upgrade instructions for Aegir, while typically describing how to update your Aegir instance to a new release of Aegir specifically, can also be used as-is to perform an upgrade of your Aegir system to the latest version of Drupal only.

In other words, follow the Aegir upgrade instructions per that link above as though you were upgrading Aegir, but retain the same version number in the steps:

The next step in the upgrade would normally be:

The upgrade steps will actually build a new Drupal platform ($HOME/hostmaster-$AEGIR_VERSION) automatically using Drush Make.

However, you can't have two directories in /var/aegir called 'hostmaster-6.x-1.1' because Drush Make will abort on this.

To work around this, adjust the 'target platform' part of this command so that the new platform directory will have a unique name:

The naming convention is arbitrary here: I am simply suffixing it with d6.22 so I understand why I did it if I forget later :)

So long as it's a unique new target platform, your Aegir instance will 'upgrade' itself by backing up and then moving the Aegir frontend onto the new Drupal 6.22 platform, then running through the rest of the upgrade/ update.php steps automatically.

Afterward, you can safely visit the old Hostmaster platform node (usually /hosting/c/platform_hostmaster) in your Aegir frontend and remove it by running the 'Delete' task. I recommend you don't simply rm -rf the old hostmaster platform manually, but let the Delete task take care of this, and other cleanup tasks, for you.

As always if you have questions or get stuck, let us know on the Community portal, on the mailing lists, the Support ticket queue or on IRC in #aegir on Freenode.

Good luck!

------

If you need a consultant to handle the tasks of installing, upgrading or troubleshooting your Aegir instance, consider reviewing our consultancy services. mig5 is a core developer of the opensource Aegir hosting system.