Our zero-touch build setup (continuous integration "lite")

We’re a software company that has a SAAS offering built on top of Drupal. We use (and LOVE) Aegir to manage instance deployments as well as development and deployment workflow. For a while, we’ve had a deployment workflow running smoothly with Aegir, based on the concepts presented by mig5 in his excellent article on the subject http://community.aegirproject.org/article-archive/drupal-deployments-wor...

However, we found that we were spending lots of time manually setting up platforms, waiting for sites to migrate, etc. It was a waste of time, and also meant that we didn’t test our progress in a pre-production environment as often as we should. I read with great interest another of mig5’s articles on continuous deployment with Jenkins, but this seemed a bit overkill for what we needed to do, and fairly complicated to set up.

So we set about creating our own zero-touch deployment approach which I’d like to share with everyone here in case it can help someone and to get your feedback. It’s quite simple, and consists of 2 drush scripts, a bash script, and a makefile.

Before you get started, it’s best to re-read mig5’s article on deployment workflows with version control and aegir, since what we’re doing is an extension of this idea. I’m also going to assume you’ve got an existing aegir platform up and running with command line access, you’ve got a test site (or sites) running on aegir, and your platform code is all stored in version control.

Start with a makefile

To begin, you’ll want to setup a makefile for your build, and make sure that it’s working correctly to build your platform. Here’s our makefile as an example. We cheated a little bit compared to the “pure” approach to makefiles, as we’ve got our entire sites/all directory stored in version control. We’ve got lots of custom and contrib modules, some of which we’ve modified, and it was easier to do it this way rather than try to set up individual repos for each module and fetch contrib ones individually in the makefile. Here's our makefile as an example. The root of our repo starts at sites/all, so this is where we will pull in all the code (cheating, using the "library" function of make, and telling make to overwrite the one file that is already there by default)

api = 2

;Grab Pressflow Core

core = 6.x
projects[pressflow][type] = "core"
projects[pressflow][download][type] = "get"
projects[pressflow][download][url] = "https://github.com/pressflow/6/tarball/master"
;Patch language.inc to avoid forcing domain name
projects[pressflow][patch][] = "http://aaa.bbb.com/files/language.patch"

;We're badly behaved children, so we're just going to grab all of our modules and everything in one swoop

libraries[all][download][type] = "git"
libraries[all][download][url] = "git@bitbucket.org:g----/xxxxx.git"
libraries[all][download][branch] = "develop"
libraries[all][download][username] = "g-------"
libraries[all][download][password] = "****"
libraries[all][destination] = ".."
libraries[all][directory] = "all"
libraries[all][overwrite] = TRUE

If, like us, you’re having a bit of trouble getting git setup correctly, make sure that you generate ssh keys with the aegir user on your primary platform, add the aegir public ssh key to your github/bitbucket repo, and then do a git clone on the command line to test everything out before you let aegir do it for you.

With this in place, you should now have an aegir setup where in the frontend, you can easily create a new platform off of your develop branch (or whatever branch you want to use for automatic deployments), and then again in the front-end, migrate your test site(s) to the new platform.

Now comes the fun part – automating this!

Here are the steps we want to automate: 1. Check if there’s been an update to our platform code 2. Create a new platform based on the updated code 3. Migrate our test site to the new platform 4. Clean up any unused platforms

Here’s the bash script that does all of these steps.

#!/bin/bash

cd /var/aegir

#Update git remote details
echo "Checking git updates"
git --git-dir=/var/aegir/repo/develop/gcapp/.git --work-tree=/var/aegir/repo/develop/gcapp remote update

#Check if there are changes to the develop branch
if ! git --git-dir=/var/aegir/repo/develop/gcapp/.git --work-tree=/var/aegir/repo/develop/gcapp diff develop origin/develop --quiet; then
echo "Updates found on remote! Building platform"
  
   # Update local repo from git, so that we can know whether future changes have occurred
git --git-dir=/var/aegir/repo/develop/gcapp/.git --work-tree=/var/aegir/repo/develop/gcapp pull
   
   GCAUTOPLATFORM=autobuild_$(date +%Y%m%d)$(git --git-dir=/var/aegir/repo/develop/gcapp/.git --work-tree=/var/aegir/repo/develop/gcapp describe)
   
   #Note, need to run drush as uid 1, otherwise platform isn't available to sites
    # Create platform
  drush --user=1 provision-save @platform
$GCAUTOPLATFORM --context_type=platform \
          --root=/var/aegir/platforms/autobuild/$GCAUTOPLATFORM \
            --web_server=@server_admin \
           --makefile='/var/aegir/makes/givco_preprod_git.make'

    # Verify settings and generate the directory with drush make, pushing to the remote server after.
  drush --user=1 provision-verify @platform_$GCAUTOPLATFORM

# Import into hostmaster front-end
drush --user=1 @hostmaster hosting-import @platform_$GCAUTOPLATFORM
   
   # Migrate test site
    /var/aegir/makes/migrate.drush @hostmaster druptest.givingcorner.com $GCAUTOPLATFORM

fi

#Cleanup old platforms

/var/aegir/makes/cleanup.drush @hostmaster /var/aegir/platforms/autobuild

In order to automate this so that it runs at a regular interval (we check every 10 minutes), as the aegir user add it to your crontab using crontab –e, and add the following line
*/10 * * * * /var/aegir/makes/autoplatform.sh

Now let’s go through what we’re doing in each of these steps:

1. Check if there’s been an update to our platform code

We keep a local version of our platform code in /var/aegir/repo/develop/ so that we can use it is as a base for comparison to see if the remote code has changed. In theory, it would have been cleaner to setup a commit hook which would then trigger our updates, but this is the “lite” version, so we really wanted to stick with stuff that could be easily integrated into a shells script ;-)

We first check for remote updates, and then perform a diff to see if there are differences between the remote and the local versions. Using the --quiet switch allows us to simply check an exit code to see if diff returns something or not before continuing. Once we know that we need to build a new platform, we also need to update our local copy of the repo so that we’re in sync with the remote version and we don’t keep triggering a build of a new platform!

2. Create a new platform based on the updated code

Now that we know that the code has been updated, we can tell aegir to build us a new platform. The makefile has all the details about how to build out the platform and will get the sources for us, so we just need to create the platform in the backend, then import it into the front-end (this part is directly borrowed from mig5’s setup) We set up an environment variable with our (automatically generated) platform name, which simplifies the process of creating the platform

3. Migrate our test site to the new platform

There’s currently no easy way to migrate a site from the command line (See http://drupal.org/node/1003536 )

But Drush script to the rescue! The following drush script takes as arguments the name of the site to migrate, and the name of the target platform, and looks up all the information needed to add a hosting migrate task to the front-end, and migrate the site on the backend. Since aegir tasks happen asynchronously, we also added a waiting period of up to 10 minutes for the platform to be available, since in the case of our automated deployment, we need to give it time to build

Here’s the drush script (remember to chmod u+x it so that it can be run from the shell)

#!/usr/bin/env drush

<?php
//Drush script to migrate a site from the command line


//Provide site name and target platform name from the command line
$site_name = drush_shift();
$platform_name = drush_shift();

//Lookup site 
$sql = "SELECT nid FROM node where title = '%s' AND type = 'site';";
$result = db_query($sql, $site_name);
while (
$row = db_fetch_array($result)) {
 
$site_nid = $row['nid'];
}

if (!
$site_nid) {
 
drush_set_error('INVALID_SITE', "Specified site cannot be found");
  exit();
}


//Lookup platform 
$sql = "SELECT nid FROM node where title = '%s' AND type = 'platform'";
$result = db_query($sql, $platform_name);
while (
$row = db_fetch_array($result)) {
 
$platform_nid = $row['nid'];
}

if (!
$platform_nid) {
 
drush_set_error('INVALID_PLATFORM', "Specified target platform cannot be found");
  exit();
}

#load full site details so we can add to migrate task
$site = node_load($site_nid);

#Confirm if platform is online, if not, wait

$timeout = 600;
$online = FALSE;
$start_time = time();
while (!
$online) {

 
//Get platform readiness
 
 
$sql = "SELECT status FROM hosting_platform WHERE nid = %d AND status = 1";
 
$result = db_query($sql, $platform_nid);
  while (
$row = db_fetch_array($result)) {
     
$online = $row['status'];
    }
   
 
//If we're not yet online, sleep for 15 seconds, unless we've already hit our timeout
 
if (!$online) {
    if (
time() - $start_time > $timeout) {
     
drush_set_error('PLATFORM_TIMEOUT', "Target migration platform was not ready within timeout period of $timeout seconds");
      exit();
    }
   
sleep(15);
  }
}

//Platform must be online, so we can continue with the migrate

hosting_add_task($site_nid, 'migrate', array(
   
'target_platform' => $platform_nid,
   
'new_uri' => $site->title,
   
'new_db_server' => $site->db_server,
));
?>

To run it, call it within the hostmaster context
/var/aegir/makes/migrate.drush @hostmaster SITE_TO_MIGRATE  TARGET_PLATFORM

4. Clean up any unused platforms

Finally, we’ll want to clean up old platforms we’ve already migrated away from, otherwise we’ll clutter up the interface (and our disk space!)

Again, there’s no easy way to do this from command line (http://community.aegirproject.org/discuss/there-way-delete-platform-comm...), so we wrote another drush script to handle it

#!/usr/bin/env drush

<?php

//Cleanup script to delete empty autogenerated platforms older than 2 days

//Provide the platform location via the command line
$platform_location = drush_shift();

$sql = "SELECT n.nid FROM hosting_platform AS p
  INNER JOIN node AS n ON (p.nid=n.nid)
  LEFT JOIN hosting_site AS s ON (p.nid=s.platform)
  WHERE s.platform IS NULL AND n.created < (UNIX_TIMESTAMP() - 172800) AND p.publish_path LIKE '%%%s%%' AND p.status = 1;"
;

$result = db_query($sql, $platform_location);
while (
$row = db_fetch_array($result)) {
 
hosting_add_task($row['nid'], 'delete');
}

?>

Again, you’ll want to call it from within the hostmaster context
cleanup.drush @hostmaster PLATFORM_PREFIX

Please feel free to provide suggestions or improvements, or move it to the handbook if it is helpful for others!

#1

Thanks for sharing Gmania :) Here is a related post you might be interested in http://community.aegirproject.org/discuss/svn-or-any-version-control-fri...