community.aegirproject.org

This article first appeared on mig5's blog on 9th June, 2011.

A while ago I wrote that I had started to use Jenkins for various purposes such as server backups, providing better notification in event of failure, yadda yadda.

Over at the Aegir project I implemented Jenkins to give us a continuous integration platform that kicks into gear either on demand or whenever one of us pushes a commit to the Drupal git repos. It does the following:

• Provisions a new virtual server (at Rackspace Cloud) via the Libcloud API
• Installs Aegir on the server
• Builds a Drupal 6, Drupal 7 and OpenAtrium platform via Drush Make
• Installs a site on each respective platform

A number of other build jobs exist, such as testing upgrades of earlier Aegir releases to latest versions, but I won't go into that here.

What several people have asked me to elaborate on, is that the combination of Jenkins, Cloud-based virtualisation and Aegir can make a very useful testing tool if you're building some sort of Drupal solution like a distribution or some sort of turnkey Drupal appliance where you want to repeatedly test the installation process (especially with a custom install profile, for example). This article will describe how we built such a thing.

Disclaimers before we start

1. I'll assume you have a server set up with Jenkins installed already, as that's out of the scope of this article.

2. The code I'm giving you uses Libcloud API, but it is very much designed for the Rackspace Cloud driver implementation. I know for a fact, through building other tools, that other provider drivers such as Linode have differing requirements (Linode, for example, requires you to also specify a datacentre to build in). Please don't contact me complaining that it didn't work with Provider X, because I'm just showing you how we use it with Rackspace. Since there's a good chance you're smarter than me, you're welcome to take the code and abstract it properly :)

3. The code is far from perfect because I'm rather crap at such things. The error handling is extremely crude.

4. I am not going to explain every line of code in this article: this is targeted towards clever people who will read the code and understand what to tweak, what assumptions to add or remove to make it build their own project. Again, this is just what (currently) works for us.

5. I am assuming the Jenkins server is running on Debian Squeeze, where things like fabric and python-setuptools can simply be apt-get installed with no fuss.

Step 1) Install libcloud tools

Libcloud will provide the API necessary for provisioning a brand new virtual server at a provider such as Rackspace Cloud.

Run these commands as root / with sudo on your Jenkins server.

You'll need python-setuptools on a Debian/Ubuntu system:

Install libcloud likeso:

Step 2) Install fabric

Fabric, a python library for communicating with servers over SSH, will run necessary commands to the new VPS to install Aegir and the rest, via SSH.

Run this command as root / with sudo on your Jenkins server.

Step 3) Upload scripts to the Jenkins server and edit

Two files provide the magic that does the whole job of building a new VPS and installing Aegir and the rest. You want these two files on the Jenkins server for the Jenkins user to use.

The .ini file is for you to edit and enter your API keys, server specs and relevant details such as your email address, trusted IP for firewall whitelisting on the new VPS, and URL to fetch your makefiles from to build your project.

The other file with the .py extension is the actual python code that Jenkins will execute to build the whole thing.

You can clone my repo containing the two files from github:

The code assumes this directory containing those files are in the home directory of the Jenkins user, e.g /var/lib/jenkins/aegir_ci

It's at this point that you want to edit the aegir_6.x-1.1_install.py file and define what to build. In my example that I'm giving you, I'm fetching three Drush make stub or 'build' files from my github repo, named drupal6, drupal7 and openatrium .build respectively.

The code that does this is the function fab_install_platform().

Thus, this code is very much specific to building your project from a makefile, the fact that the makefile is fetchable over HTTP (per the ini config file, it fetches from my github repo) and also makes some assumptions on what install profile to use when installing the site.

The code that installs the site is the function fab_install_site(), and takes two arguments: the platform per the above (e.g drupal6, drupal7 or openatrium), and the install profile name (default, standard or openatrium in this example). So this is where you want to change the install profile name and only install whatever platforms you want, based on the name of your makefile.

Step 4) Add a new Jenkins job

Add a New Job of type 'free-style software project'.

In the combobox 'Add Build step', choose 'Execute shell'.

In the text field that appears, enter this line:

Choose any other settings you'd like in the job, such as notifications if a build fails, whether you want to automate this build on a scheduled basis (like a cron), and so on.

Step 5) Execute a Build

After you've saved the job, simply click 'Build Now' in the left sidebar and watch the Console Output to see the new server get provisioned, your Aegir installed and your distro or custom Drupal app built and a test site install processed.

Blowing it all away

My example code has a snippet at the bottom that completely destroys the VPS at the end. This is because our Aegir project builds on a scheduled basis, and that includes times where I'm asleep: I don't want these cruft VPS lingering around costing me money.

You may wish to prevent the auto-destruction so that you can login and take a look around, especially where there are issues. To do so, just comment out conn.destroy_node(node) at the bottom of the script.

To login to Aegir, either find the one-time login URL in the Console Output of the job itself, or check your e-mail, where you should have received the link in the welcome message.