The source code of Aegir is located on Drupal.org, as with the issue queues and release nodes.

Aegir is also made up of several separate components designed to work together, that are grouped in two main git repositories: hostmaster and provision. This page details those locations.

Newcomers to git should probably consult our git crash course while people already familiar with git can have a reminder on how to use it within aegir through the basic git workflow page.

Repositories locations

There are the URLs you can use to pull the repositories:

git clone http://git.drupal.org/project/provision.git
git clone http://git.drupal.org/project/hostmaster.git

Developers with push access to the repositories should pull via SSH.

git clone ssh://[username]@git.drupal.org/project/provision.git
git clone ssh://[username]@git.drupal.org/project/hostmaster.git

There is a web interface for your perusal at:

• http://drupalcode.org/project/provision.git
• http://drupalcode.org/project/hostmaster.git

The git repositories were hosted for a while on Koumbit's servers, but were migrated back to Drupal.org during the great git.drupal.org migration. Do not use git.aegirproject.org as it doesn't have the most recent commits and may be shutdown at any time.

Clone a repo

Let's take hostmaster:

If successful, you will have a hostmaster directory at the last state of development ('HEAD' in CVS speak, usually 'master' branch in git speak, though in our case this is now 6.x-1.x).

Check the status of the local git repository:

See the commit messages:

Check all 'releases'

Create a new branch for your local development:

Create a new branch for development, based on an existing remote branch or tag

So say you have some work you want to do on the hostmaster module. You first need to clone the repository:

That will create a hostmaster directory with the whole history of the project, and a checkout of the "master" branch (which is the equivalent of CVS's "HEAD" in git).

You make some modifications to the code to fix a bug, and then, you make a commit. That commit will only be local for now, that's the way git works:

Note that contrarily to subversion or CVS, you need to explicitly mention which file(s) you want to commit. You can also use the -a flag to commit all changes.

If you have write access to the repository, you can now push your changes straight into git. However, the git:// url above is readonly, so you will want to use SSH to push your changes. For that, we will have had to add your SSH public key to our repositories. You may also want to add a special "remote" repository to push your changes. You will need to do this only once (per repository):

You can now push your changes to that repository.

If you do not have write access to the repository, you can send the Aegir developers patches by submitting them for review in the Issue queues (see the patch contributor guide on drupal.org for more details) or by uploading them to your personnal sandbox.

Aegir development makes a heavy use of branches to keep development stable. We use feature branches to do large development work and have release branches to keep stable releases maintained while development happen on the development branches.

1. Summary

• The current development branch is: 7.x-3.x
• The current stable branch is: 6.x-2.x

2. Branch naming conventions

All branches are either development branches, stable branches or feature branches. Development and stable branches have the same naming convention: 6.x-2.x or 7.x-3.x. The only different between the two is that a stable branch is the branch where stable releases are published from.

Note: since the great git.drupal.org migration, we are back on Drupal.org and have been forced to change some of our naming convention, especially to include the core release number in the tag (so 6.x-2.x), which is a source of confusion for our provision module which supports any Drupal core version. See this issue on drupal.org for followup.

2.1. Main development branch

The main development branch is created after a fully stable release is published. For example, the 6.x-2.x branch has been created when the 1.0 release was published. It is a development branch until it is considered feature-complete and goes through the alpha/beta/rc release cycle, at which point it will be declared a stable branch and a new development branch will be created (7.x-3.x at that point).

The development branch is not stable and should not be used in production environments.

Note that we have dropped the master branch as it is not well supported on Drupal.org for releases.

2.2. Stable release branches

Stable releases are performed from stable release branches. A branch is declared stable only after it went through the complete alpha/beta/rc release cycle, as detailed above..

Once the branch is declared stable, new releases are quickly done as "hotfixes", without intervening alpha/beta/rc releases. So for example, 6.x-1.0 will come after 6.x-1.0-alpha1, 6.x-1.0-beta1, 6.x-1.0-rc1 at least, and then be followed directly by 6.x-1.1.

Releases on the same stable branch are API-compatible with each other (so 1.1 is compatible with 1.0). The upgrade path is supported between major stable releases (so you can upgrade from 1.0 to 1.1 or 2.0).

Special case: during the 0.x release cycle, releases were not API-compatible with each other, ie. 0.4 was not compatible with 0.3, and you could barely upgrade from 0.3 to 0.4.

Only one stable branch is maintained at a time.

2.3. Feature branches

More complex development goes into development or "feature" branches, to avoid making master too unstable. Feature branches must be prefixed with "dev-" and use the issue number (if available or relevant) and a short name. So for example, there could be work on a dev-12345-mybug branch or dev-dns branch for a larger feature (DNS, in that case) which doesn't fit in a single issue.

Feature branches should be based of a known stable baseline as much as possible. For example, you should branch off the last alpha release if you want to make merging with other dev branches and master easier. See this article for a thorough discussion on this practice

3. Tag naming convention

Every release is tagged with a 6.x-x.y tag (e.g. 6.x-0.4, 6.x-0.4-alpha15 or 6.x-1.0). Those tags are usually laid on the stable branches, except for alpha/beta/rc releases which are laid on the development branch.

4. CVS and git history

The CVS tags and branches are still accessible in git and will be kept there for posterity. They respect the traditional Drupal.org naming convention.

They have also been converted to the new git.drupal.org conventions during the great git.drupal.org migration.

Since that migration (february 2011), history has been rewritten to clean up the authors in the history.

After thorough discussions where we weighted the pros and cons of profiting from the Great Git Migration, I think it's pretty clear that we have a great opportunity to migrate our source repositories back to Drupal.org. We will get:

• more visibility - people will find us through drupal.org easily
• more consistency - versions in the issue queue matching the real releases
• even more consistency - people will find the source better from drupal.org
• and then more consistency - usage stats will work again!

We will be able to:

• host our module and theme within the install profile - the major reason why we migrated away back then
• use whatever naming convention we want for tags and branches - although only some will be used for releases for now, see this issue for follow-up
• keep our history - the git team was nice enough to clone directly from our repositories (although see below, we're taking the opportunity of the migration to clean up our history)

What it means for you

This does have an impact on your existing repositories. Since we migrated to git.drupal.org, the repositories have changed location. The other thing is that the commit IDs have changed, and you will need to clone new checkouts before you can work with them effectively.

So in short:

• repos will move to git.drupal.org
• you need to clone again
• make sure your identity is set correctly

URL changes

So the repositories have moved. Everyone has to fetch the source code from git.drupal.org now. At some point in the future, Koumbit will close git.aegirproject.org. Concretely, Koumbit will keep the mirror running readonly for a while just to be on the safe side, but you shouldn't expect git.aegirproject.org to work and should switch URLs, as detailed below.

This documentation is taken from the Drupal.org git workflows documentation.

Readonly repositories URL change

The readonly access will change from:

to:

See also the git patch maintainer guide.

Core committers repositories URL change

The read/write access will change from:

to:

... notice how you will need to set your Drupal.org username in the URL. Drupal.org has good documentation on how authentication works on git.drupal.org. See also the project maintainer git guide.

Sandboxes

The migration to Drupal.org also introduces sandboxes, something we didn't have before, and which allows you to host your own branches of our modules (even if you're not a core committer) or contrib/test modules, directly on Drupal.org. See the sandbox collaboration guide for more information on that.

What if I pull/push from/to the wrong one?

It could happen before/during/after the migration that you pull or push from or to the wrong repository. git can rebase with the new repository, but unfortunately, the old commits will be mingled with the new ones.

To be able to push to the new repositories, you will need to follow instructions below to cleanup your repositories.

Rewriting history

I have taken the liberty of rewriting our collective history. There were a lot of commits with inconsistent authors: some had invalid emails, some had no real names, some had multiple different email addresses for the same person. I narrowed the list down, through extensive detective work, to a complete list of 14 distinct authors.

I have rewritten all commit authors that didn't seem right to follow a consistent pattern:

The firstname/lastname is the combination you have used in certain commits. It's not the most popular way (ie. most commits had the nickname instead of first/last), but it seems the most logical one.

The email was your most often used email or the one you have registered as such at drupal.org, if you didn't have an email address specified (in case you committed your code back in the days of CVS). (There are 6 commits from someone with a box at ovh.net that I could track, and that I can only guess are Miguel's, but I'm not sure so I have let those one fly. :)

You should make sure you have your identity set straight in your git configuration, especially your email address, if you want your contributions to be properly credited in the future on Drupal.org. In short, this means:

Again, see the manual for more information, especially how this maps to your Drupal.org account.

A fresh clone is necessary!!!!

Note that, because we rewrote history, you will need to clone your repository from new remote. So yes, contrarily to what was documented here before, you will need to clone from scratch.

We understand this may be annoying if you work on a bunch of branches or have local branches you want to keep. Unfortunately, our original tests didn't cover all use cases and were misleading in thinking we could just rebase existing repositories.

Other documentation

People unfamiliar with git or specifically git on Drupal.org should read the growing git handbook on Drupal.org.

People that are familiar with git should contribute to that manual. :)

Under the hood

The history rewrite was performed through git-fast-export and git-fast-import magic, with a fairly simple perl script of my own. I have cloned the current repo, filtered it and published a new version for git.drupal.org to pull. You can see the results here:

The exact calling sequence is this:

mkdir export orig
cd export
git init --bare hostmaster.git
cd ../orig
git clone --bare git://git.aegirproject.org/hostmaster
cd hostmaster.git
# look at the authors in the original repo
git fast-export --all |  grep -a '^author [^<]* <[^>]*> [0-9][0-9]* [+-][0-9][0-9][0-9][0-9]$' | sed 's/[0-9][0-9]* [+-][0-9][0-9][0-9][0-9]$//' | sort | uniq -c
git fast-export --all | perl /home/anarcat/bin/rewrite_authors.pl | ( cd ../../export/hostmaster.git ; git fast-import )
cd ../../export/hostmaster.git
# look at the authors in the new repo
git fast-export --all |  grep -a '^author [^<]* <[^>]*> [0-9][0-9]* [+-][0-9][0-9][0-9][0-9]$' | sed 's/[0-9][0-9]* [+-][0-9][0-9][0-9][0-9]$//' | sort | uniq -c
# rinse, repeat until the mapping is right
# push repositories online
git remote add origin ssh://gitosis@git.aegirproject.org/export/hostmaster.git
git push --all
git push --tags
# repeat with provision

This is the script source, without all the mappings:

#! /usr/bin/perl -w

%authors = ( 
    'anarcat ' => 'Antoine Beaupré ',
   # ... 
);

sub remap {
    my $a = shift;
    if ($authors{$a}) {
        return $authors{$a};
    } else {
        return $a;
    }
}

while (<>) {
    s#^author ([^<]+ <[^>]+>) ([0-9]+ [+-][0-9][0-9][0-9][0-9])$#'author ' . remap($1) . ' ' . $2#e;
    print;
}

Migration checklist

1. serve exported repositories to migration team done!
2. update the repositories with change authors done!
3. the great git migration done! git.aegirproject.org is now readonly, commits are pushed to git.drupal.org and releases are performed on drupal.org
4. update documentation in handbook in progress!
   • repositories locations done
   • crash course done
   • basic workflow done
   • release process needs review - updated for release nodes stuff, will check workflow during the next release
   • advanced workflow: branch naming conventions in progress - see 0.4-rc2 release coordination
   • migration of the install docs into the handbook work in progress
5. update the home page links done - I have changed the link in the body from git.aegirproject.org to drupal.org provision/hostmaster project pages, and pointed the "Get the source" link in the bottom to the install manual instead
6. update makefiles and files in docs/* in provision to fetch from git.drupal.org done!
7. make at least one other release done - 0.4-rc2 release coordination
8. turn off git.aegirproject.org done

This section deals with providing coherence and understanding of various aspects of Aegir - what it is, and what it is not.

It tries to explain some of the concepts with which Aegir is developed for and how all the bits hook together.

@TODO - this document is out of date and needs updating.

This page documents all the different terms used when referring to components of the Hostmaster system, and how the different entities relate to each other.

Front End

The user interface used to administrate your sites. The front end is provided by the Hostmaster install profile, and the Hosting contributed module. It defines a complete Drupal based data model for managing the various aspects of your installation.

Entities

These entities are used primarily in the front end to manage the configuration and data. Upon calling the back end, the system breaks down the properties of these entities to be passed as options for the command line, so the back end only has a flat data structure to work with. During this process, all relationships are automatically retrieved, making it a one step process for the developer.

Client

The person or group that runs the site. This information is usually required for billing and access purposes, to assure that only certain people are able to view the information for sites they run. If you do not intend on having more than one client access the system, you will not need to create any additional clients for your purposes.

Site

An instance of a hosted site. It contains information relating to the site, most notably the domain name, database server and platform it is being published on. A site may also have several aliases for additional domains the site needs to be accessible on.

Task

The mechanism whereby Hostmaster keeps track of all changes that occur to the system. Each task acts as a command for the back end, and contains a full log of all changes that have occurred. If a task should fail, the administrator will be notified with an explanation of exactly what went wrong, and how to fix it.

Platform

The file system location on a specific web server on which to publish sites. Multiple platforms can co-exist on the same web server, and need to do so for upgrades to be managed, as this is accomplished by moving the site to a platform hosting an updated release. Platforms are most commonly built for specific releases of Drupal.

Server

The server that provides various services.

Service

The service that runs on the server. This might be HTTP or a Database server. Each web server hosts one or more platforms, which act as publishing points for the hosted sites. If you are not intending to use Hostmaster in a distributed fashion, you will not need to create additional web servers for your purposes. Most web servers and database servers are on the same machine, but for performance reasons external database servers might be required. It is not uncommon for one database server to be shared amongst all site instances. If you are not intending to use an external database server, or multiple database servers, you will not need to create any additional database servers for your purposes.

Service type

The type of service that runs on the server. In the example of 'HTTP' as a service, a service *type* is Apache or Nginx, for example. MySQL is a *type* of service 'Database'.

Package

An installable component for a site. Hostmaster keeps tracks of all the modules, themes and install profiles installed across your entire network. These packages most commonly link to their project on drupal.org, but might not in the case of custom modules. Users are not able to create nodes of this type, as they are generated automatically during verify and sync tasks.

Package Release

The release of a specific package. Each package on your network has at least one installed release, and each platform (including Drupal itself) may have several releases installed across your network.

Package Instance

Where a package release has been installed. A package release may be installed in multiple places on a platform, and sites may have access to multiple versions of packages. This entity also tracks which modules are enabled, and what version of the schema (if any) they have installed.

Task types

These are "hosting commands" that the server can accomplish. These are mapped to back end commands by the task queue processor.

Verify platform

Verify that a platform is correctly installed. Automatically created when new platforms are added, to ensure that they are capable of having new sites installed on them. Operates on: platforms

Import sites

Import existing sites onto a platform. Operates on: platforms

Install site

Install a new site. This is automatically generated when a site node is created. Operates on: sites

Sync site

Synchronize changes to the site record with the back end. Automatically generated on editing of the site node. Operates on: sites

Backup site

Generate a backup of an existing site. This backup contains everything needed to run the site. Operates on: sites

Restore backup

Restore a site to a previous backed up version. Operates on: sites

Disable site

Disable a running site. This is commonly used in the case of non-payment from clients. Operates on: sites

Enable site

Enable a disabled site. The opposite of Disable. Operates on: sites

This is an incomplete list, and will be updated as more tasks are added

Queues

The front end uses and maintains several queues, which are then processed through the drush.php hosting dispatch command. More queues can be added by optional contributed modules, and are often used to provide regularly scheduled back end commands, such as backing up sites.

Task Queue

The primary and most important queue. Whenever a change is made to the data set, the front end creates a "Task" node. These task relate to such things as installing new sites, regenerating the configuration files of sites, verifying that a platform has been correctly configured and importing existing sites on a newly added platform.

Cron Queue

This queue is used to manage the regular execution of the cron process required by most sites. It is configurable with a frequency you want all the sites to be cronned within. This is commonly 1 hour, but might be higher or lower. Higher frequency will cause higher system load. This command will iterate through all enabled sites, and batch the cron calls, based on how many sites are available, and how frequently it is running.

Backup Queue

This queue operates in the same way as the cron queue, in that it iterates through all installed sites, based on a configurable frequency. This executes the same functionality as requesting a site backup through the user interface. You are able to revert back to a previous backup of your sites.

Statistics Queue

This queue operates on the same mechanism as the cron queue, and alows you to retrieve statistics of your installed sites. This will retrieve such metrics as number of registered and active users, and number of posts / comments, which will then be viewable on the front end.

Back End

The back end is a command line interface provided by the Provisioning Framework, of all the tasks it is capable of performing.

It is designed to operate separately from the front end, so that you can have multiple back ends on one or more servers.

The back end handles the server level configuration of the system, such as creating new databases and virtual host records and restarting the apache process for new installations to take effect.

When the queues are processed by the front end, these are turned into calls to the back end, for whichever server they happen to be on. The back end communicates to the front end through a very simple mechanism, whereby it returns a serialized string containing state information and log messages / error return codes.

This wiki page documents the Aegir architecture in schematic form. It is intented to complement a related page on the typical file system structure on an Aegir installation.

Click on image for a larger version.

This page documents the typical file system structure on an Aegir installation. It is intended to complement a related wiki page on Aegir Architecture. The following paths are based on an Aegir 0.4-x installation and assume that all directories are within the /var/aegir folder.

Scripts and Configuration Files

Path	Notes
/backups	site-specific tar balls containing a database dump and folders under path/to/platform/sites/example.com
/config
/config/server_master
/config/server_master/apache
/config/server_master/apache/conf.d	non-aegir or non-drupal virtual hosts files
/config/server_master/apache/platform.d	contains .htaccess information for each aegir platform
/config/server_master/apache/vhost.d	apache virtual host files for aegir platforms
/config/server_master/apache/vhost.d/aegir.example.com	virtual host file for aegir web site - specifies path to platform directory and site database settings (so that database credentials are not exposed directly in site settings.php file)
/config/server_master/apache/vhost.d/site-1.com	virtual host file for deployed web site
/config/server_master/apache/vhost.d/site-2.com
/config/server_master/apache/vhost.d/site-3.com
/drush	drush folder
/drush/drush.php	drush script
/.drush	drush extensions and server, platform and site aliases
/.drush/drush_make	drush_make project folder - used for building web sites from .make files
/.drush/provision	provision folder
/.drush/server_master.alias.drushrc.php	settings for the master server where the main aegir database, hosting platform and aegir site reside
/.drush/platform_hostmaster.alias.drushrc.php	settings for the hostmaster platform on which the aegir site is based
/.drush/hostmaster.alias.drushrc.php	settings for the aegir site
/.drush/platform_platform1.alias.drushrc.php	settings for platform1 on which site-1.com is based
/.drush/site-1.com.alias.drushrc.php	settings for site-1.com

Hostmaster Platform and Aegir (front-end) Site

Path	Notes
/hostmaster-0.x	hostmaster platform
/hostmaster-0.x/profiles
/hostmaster-0.x/profiles/default
/hostmaster-0.x/profiles/hostmaster	hostmaster profile
/hostmaster-0.x/profiles/hostmaster/modules
/hostmaster-0.x/profiles/hostmaster/modules/admin_menu
/hostmaster-0.x/profiles/hostmaster/modules/hosting	hosting module
/hostmaster-0.x/profiles/hostmaster/modules/install_profile_api	install_profile_api – facilitates provisioning of sites based on a non-default profile
/hostmaster-0.x/profiles/hostmaster/modules/jquery_ui
/hostmaster-0.x/profiles/hostmaster/modules/modalframe
/hostmaster-0.x/profiles/hostmaster/themes
/hostmaster-0.x/profiles/hostmaster/themes/eldir	eldir theme – provides Aegir front end look and feel
/hostmaster-0.x/profiles/hostmaster/hostmaster.profile	profile file – used in site provisioning to configure a drupal database
/hostmaster-0.x/profiles/hostmaster/hostmaster.make	make file – used to include modules, themes, libraries etc. from various sources
/hostmaster-0.x/modules
/hostmaster-0.x/themes
/hostmaster-0.x/sites
/hostmaster-0.x/sites/aegir.example.com	aegir web site folders

Deployed Platforms

Note: the directory /platforms is optional but can be useful to separate deployed platforms from directories for scripts, config files and hostmaster platform.

Path	Notes
/platforms
/platforms/platform-1
/platforms/platform-1/profiles
/platforms/platform-1/profiles/default
/platforms/platform-1/profiles/custom-profile
/platforms/platform-1/profiles/custom-profile/modules
/platforms/platform-1/profiles/custom-profile/themes
/platforms/platform-1/profiles/custom-profile/custom.profile	profile file – used in site provisioning to configure a drupal database
/platforms/platform-1/profiles/custom-profile/custom.make	make file – used to include modules, themes, libraries etc. from various sources
/platforms/platform-1/modules
/platforms/platform-1/themes
/platforms/platform-1/sites
/platforms/platform-1/sites/site-1.com
/platforms/platform-1/sites/site-1.com/modules
/platforms/platform-1/sites/site-1.com/themes
/platforms/platform-1/sites/site-1.com/files
/platforms/platform-1/sites/site-1.com/settings.php	site-specific drupal configuration file
/platforms/platform-1/sites/site-1.com/drushrc.php	site-specific aegir-specific configuration file
/platforms/platform-1/sites/site-2.com
/platforms/platform-1/sites/site-3.com
/platforms/platform-1/sites/site-n.com
/platforms/platform-2
/platforms/platform-3
/platforms/platform-n

Platform permissions

Path	Directory	File	Notes
./*	aegir:aegir drwxr-xr-x	aegir:aegir -rw-r--r--	The webserver has no business writing or moving files in the Drupal codebase.
Inside ./sites/example.com:
drushrc.php		aegir:aegir -r--------	Web server shouldn't be able to read drushrc.php, it's not a component of the Drupal site.
settings.php		aegir:www-data -r--r-----	Web server can read this file, but otherwise tight control over this file which can contain sensitive information.
libraries modules themes	aegir:aegir drwxrwsr-x	aegir:aegir -rw-r--r--	Because of the sticky bit (s) on the parent directory, each child directory will inherit attributes of the parent. The attribute that is consistently inherited is the group. This means that a developer in the "aegir" group can add files which will retain the "aegir" group ownership of the parent directory.
files private	aegir:www-data drwxrws---		Aegir sets these directories with a sticky bit (s) so that under certain conditions new folders and files will inherit parent permissions. There are only a few cases where this happens though.
files/* private/*	www-data:www-data drwxr-sr-x	www-data:www-data -rw-r--r--	The permissions shown here are how files and directories created by www-data will look. When verifying a platform, Aegir won't "correct" these files and directories to match the parents. If you have trouble with permissions/ownership on these directories, you can safely run the following commands (in this case on the files directory): # chown -R aegir:www-data /path/to/site/files/* # chmod -R 775 /path/to/site/files/*

Each of the major entities managed by Aegir (namely servers, platforms and sites) are represented by nodes in the front end, which in turn generate Drush 'named contexts' in the backend. Named contexts are similar to what Drush 3.0 refers to as 'site aliases', but are more structured and cover more than just sites.

Whenever you create or edit a node on the front end, Aegir will create or update the named context on the backend, and any tasks run on the entity will be run on that named context, ie:

drush @server_master provision-verify

All of the actual work related to managing your system is done by the backend, and the front end primarily serves as a way to manage these aliases. You can very easily use the backend on it's own if you so choose.

Creating or updating named contexts.

Contexts are created and updated through the provision-save command. The format of the command is :

drush provision-save @contextname --context_type=type --property=value --other_property=value

Every time this command is called, it will update the named context and only modify the properties that have been specified. It is important to note that the provision-save command never has a site alias or context name before the command, but requires you to specify which context you are modifying as the first argument.

Deleting named contexts.

To delete an existing named context, you simply need to pass the 'delete' option to the provision-save command, ie:

drush provision-save @contextname --delete

Using named contexts

Named contexts are used in 3 ways in provision: as aliases, arguments or options.

Aliases : Aliases appear to the left of the drush command, and are making use of the Drush 'site alias' functionality.

# Verify the site represented by @example.com
drush @example.com provision-verify 

Options : Certain options to provision-save require you to pass the alias of another named context of a specific type, and uses this information to build relationships between objects.

 # generate a new site context in the directory represented by @platform_test
 drush provision-save @example.com --context_type=site --uri=example.com --platform=@platform_example

Arguments : A select few provision commands require you to pass the alias of another specific type of named context as an argument.

# Migrate the site to the platform represented by @platform_newversion
drush @example.com provision-migrate @platform_newversion

Context types

There are (currently) 3 different entities represented as named contexts in the aegir backend. You determine which type of entity you are working with by specifying the context_type option to the provision-save command. You dont need to specify it every time, but it is best to always explicitly specify it, just in case.

Because the namespace of all contexts is shared, they each have different naming conventions to avoid collisions. There are certain named contexts of each type which are always present in the system.

Each of the context types have different semantics and properties, and certain provision commands will only operate on certain types of entities.

Server contexts

Naming guideline : Server with a hostname of server.example.com becomes @server_serverexamplecom.

Special contexts : @server_master always refers to the local server where aegir is installed.

Required properties :

• remote_host : This should always be set to the FQDN of the server being managed.

Services : Servers are the only entities which have services associated to them. Services are things such as http, db or dns.

Each service may select one service_type, which chooses which implementation to load for this service. Examples of these are the 'mysql' implementation of the 'db' service, or the 'apache' and 'apache_ssl' implementations of the http service.

All services that are found by Aegir are enabled, but unless you specifically select one of the implementations, the 'null' service type is loaded. Each service type has additional required properties and values that can be passed to it, but describing them all is outside of the scope of this document.

Important commands :

• provision-verify : Verify that the configuration is correct and fix them if they aren't.

Example :

# Enable a new remote server hosting sites with apache on port 8080
drush provision-save @server_webexamplecom --context_type=server \
  --remote_host=web.example.com \
  --http_service_type='apache' \
  --http_port=8080

# Verify that the settings are correct
drush @server_webexamplecom provision-verify --debug

Platform contexts

Platforms are how aegir refers to a specific copy of Drupal checked out on a file system. Platforms are hosted on web servers and sites are hosted on platforms.

Naming guideline : A platform in /var/aegir/codebase-6.1 becomes @platform_codebase61 , but this is not strictly enforced. We only use that naming convention when we do not have a node representing the platform available. If a node is available, we use the title of the node to generate the context name.

Basically, you can name your platform anything but it should still be prefixed with 'platform_'.

Special contexts : There are no special contexts per se, but the hostmaster-install and hostmaster-migrate commands generate aliases for the hostmaster platform in the form of '@platform_hostmaster$version'. If the front end is present, one or more of these contexts are usually available too.

Required properties :

• root : The absolute path to a Drupal directory, generally in the form /var/aegir/platformdir.

Optional properties :

• web_server : The name of a server context that provides the http service. The default is to @server_master
• makefile : The absolute path or url to a drush make makefile. If the root directory does not exist and the makefile property does, provision will call drush_make to attempt to create the directory.

Important commands :

• provision-verify : This will ensure that all the files are the correct permissions, and are all present on the correct remote servers. Also rebuild package database. Run this command whenever the contents of the platform changes.

Example :

# New open atrium platform in /var/aegir/atrium-head that will be built from a remotely hosted
# makefile and published on an external web server.
drush provision-save @platform_atriumhead --context_type=platform \
        --root=/var/aegir/atrium-head \
        --web_server=@server_webexamplecom \
        --makefile='http://omega8.cc/dev/atrium_stub.make.txt'

# Verify settings and generate the directory with drush make, pushing to the remote server after.
drush provision-verify @platform_atriumhead

Site contexts

Naming guidelines : A site with the url 'site1.example.com' becomes @site1.example.com. The context name is always the url.

Special contexts : @hostmaster always refers to the site running the hostmaster front end. This helps with debugging and makes it more easily scriptable.

Required properties :

• uri : The FQDN of the site. should match the context name.
• platform : The name of a platform context that the site will be hosted on.

Optional properties :

• db_server : The name of a server context which provides the db server. defaults to @server_master.
• client_email : The email address to send the login information for the admin user.
• profile : The installation profile the site is running. dependent on the platform supporting it.
• language : The ISO language code of the site. dependent on the platform and profile supporting it.

Important commands :

Almost all of the provision commands are related to sites, so this is just the very core of them. Check drush help for more.

• provision-install : Install the site represented by the context. Just creating the context doesn't install the site yet.
• provision-verify : Verify the site is running correctly, and ensure the files are all published to the correct servers.
• provision-delete : Delete the site and all it's information. The context will still be there.

Example :

# Set up a new site to be installed on the open atrium platform (hosted on a remote server),
# using an external database server , with the openatrium profile and localize it into spanish.
drush provision-save @atrium.example.com --context_type=site \
        --uri=atrium.example.com \
        --platform=@platform_atriumhead \
        --db_server=@server_dbexamplecom \
        --client_email=client@example.com \
        --profile=openatrium \
        --language=es

# Install the new site to the specifications requested above.
drush @atrium.example.com provision-install --debug

# Verify the site to make sure everything is ok.
drush @atrium.example.com provision-verify --debug

Importing contexts into the front end

Hostmaster provides a drush command which will attempt to generate or updates front end nodes that match your defined contexts. To import a context into the front end, just run :

drush @hostmaster hosting-import @contextname

This will import that context, and every context it depends on into the front end. This is useful for integration with other systems such as the puppet-aegir module for Puppet, which can add newly configured servers directly to the front end.

So we've been talking seemingly forever about possible queuing system instead of our crappy "PHP/Vixie Cron based" one, but we have yet to document pro/cons of possible alternatives. Here are the goods (or the beginning of em).

Note that this document was written before (in november 2009) the current multiserver implementation was designed and completed.

Current design

Tasks are basically serialized in a MySQL database and polled at regular interval by a cron job, running drush hosting dispatch, which bootstraps Drupal, fetches the object, and fires up provision commands.

The possibility here is to have another dispatcher that would always run, and maybe not even talk to the mysql database, to fire up the provision commands.

Another requirement is that the queuing system may be network-aware. We want multi-server support and that is a critical issue for the next release (0.4 at the time of writing). One thing to consider is how we pass files around...

Roll our own

Pro:

• full control over the API, functionality and whatnot (ie. function of manpower invested)

Con:

• rolling our own is hard! let's go shopping!

Meta: AMQP vs JMS

So yeah, there's a standard in development for all this stuff: http://www.AMQP.org/ . There is also a Drupal module being developed to work with it: http://drupal.org/project/amqp

JMS on the other hand is the "Java Messenging Service", that creates a language- (as opposed to network-) based API.

Maybe we should just not care about which messaging system you use in the backend and allow our frontend to talk to any backend, with the stupid cronjob mysql poller as a default...

Gearman

Pros (mostly out of the home page):

• supports PHP in frontend and backend
• Multi-language - There are interfaces for a number of languages, and this list is growing. You also have the option to write heterogeneous applications with clients submitting work in one language and workers performing that work in another.
• Flexible - You are not tied to any specific design pattern. You can quickly put together distributed applications using any model you choose, one of those options being Map/Reduce.
• Fast - Gearman has a simple protocol and interface with a new optimized server in C to minimize your application overhead.
• Embeddable - Since Gearman is fast and lightweight, it is great for applications of all sizes. It is also easy to introduce into existing applications with minimal overhead.
• No single point of failure - Gearman can not only help scale systems, but can do it in a fault tolerant way.
• Production ready - ran on digg, yahoo, livejournal, ...
• Gearman uses an efficient binary protocol and no XML. There's an a line-based text protocol for admin so you can use telnet and hook into Nagios plugins.
• Persistent queues are available using mysql, memcached or sqlite
• There's pretty good PHP support and it's easy to get up and running quickly.

Cons:

• "somewhat higher latency, signal-and-pull architecture"
• The system makes no guarantees. If there's a failure the client is told about the failure and the client is responsible for retries.
• gearman-specific binary/network protocol
• Notice how sixapart.com maintains both gearman and the shwartz, below

OHLOH:

• Mostly written in Perl
• Increasing year-over-year development activity
• Established codebase
• Few source code comments

Resque

Pro:

• redis-based (therefore somehow language agnostic deep in there, but certainly network-centric)
• github's (aka: it's got to work at least somewhere in production)

Con:

• ruby-centric
• I haven't actually looked at the thing or tested it

Active MQ

Apache ActiveMQ:

Pros:

• Supports a variety of Cross Language Clients and Protocols from Java, C, C++, C#, Ruby, Perl, Python, PHP
• Supports many advanced features such as Message Groups, Virtual Destinations, Wildcards and Composite Destinations
• Supports pluggable transport protocols such as in-VM, TCP, SSL, NIO, UDP, multicast, JGroups and JXTA transports
• REST API to provide technology agnostic and language neutral web based API to messaging
• Persistent

Cons:

• uses too many acronyms on the frontpage (really.)
• uses a lot of Java jargon
• production-ready?

OHLOH:

• Mostly written in Java
• Large, active development team

RabbitMQ

RabbitMQ

Pros:

• "high reliability, availability and scalability along with good throughput and latency performance that is predictable and consistent"
• "based on the emerging AMQP standard"
• in debian
• Drupal module - AMQP
• apparently very fast, according to dellintosh

Cons:

• erlang? do you know erlang? (Arguably not really necessary, since all publishers and consumers can be written in other languages that have AMQP support)

OHLOH

• Mostly written in Erlang
• Large, active development team
• Well-commented source code

Open Messenging Queue

https://mq.dev.java.net/overview.html

Pros:

• SOAP/HTTP interface
• Scalable

Con:

• Java/C specific

The Schwartz

The Schwartz

Pros:

• Perl (come on, perl is everywhere... isn't it?)
• "reliable job queue"
• has retry
• "lightweight"

Cons:

• Perl (aka write only)
• "library, so some assembly required"
• "light on documentation"
• Notice how sixapart.com maintains both the shwartz and gearman, above

Open AMQ

Open AMQ

Pros:

• Multi-language (but not PHP)
• "learn in a day or so" then "another day or two for wireAPI", development will take "some weeks"...
• "remote admin tools, one-line failover, instant federation, protection against slow clients, detailed logging"
• AMQP standard
• "Linux, AIX, Solaris, Mac OS/X, other UNIX"

Cons:

• "for C/C++ and JMS"

OHLOH

• Mostly written in C
• Well-commented source code
• Short source control history
• Only a single active developer
• Apache Software License may conflict with GPL
• Apache License 2.0 may conflict with GPL

Beanstalkd

http://kr.github.com/beanstalkd/

Drupal Module for Drupal 6 and 7 is at http://drupal.org/project/beanstalkd

Pros: * used in production ("causes" application in facebook)

Cons:

• C

http://www.ohloh.net/p/beanstalkd

• Mostly written in C
• Large, active development team
• Few source code comments

SimpleMQ

http://code.google.com/p/simple-mq/

Pros:

• persistent or in-memory

Cons:

• java-only.

Zero MQ

http://www.zeromq.org/

Pros:

• fast
• lightweight messaging implementation.
• supports different messaging models.
• already very fast. We're getting 13.4 microseconds end-to-end latencies and up to 4,100,000 messages a second today.
• very thin. Requires just a couple of pages in resident memory.
• provides C, C++, Java, Python, .NET/Mono, Ruby, Fortran, COBOL, Tcl, Lua and Delphi language APIs.
• supports different wire-level protocols: TCP, PGM, AMQP, SCTP.
• runs on AIX, FreeBSD, HP-UX, Linux, Mac OS X, OpenBSD, OpenVMS, QNX Neutrino, Solaris and Windows.
• supports i386, x86-64, Sparc, Itanium, Alpha and ARM microarchitectures.
• fully distributed: no central servers to crash, millions of WAN and LAN nodes.

Cons:

• no PHP whatsoever
• production?

OHLOH:

• Mostly written in C++

Jenkins

There's a good vibe around using Jenins (formerly known as Hudson) in place of cron for those kind of things. In our scenario, Aegir would queue up jobs to Jenkins, which would take care of dispatching them to remote servers.

Pros:

• supports tons of different notification modes
• scales well
• supports remote servers

Cons:

• application-specific (unclear?) protocol to queue up tasks
• Java (so a bit heavy)

OHLOH: * has had 63,860 commits made by 708 contributors * representing 893,049 lines of code * is mostly written in Java * with an average number of source code comments * has a well established, mature codebase * maintained by a very large development team * with decreasing year-over-year commits

Others to evaluate?

• Amazon's SQS. The latencies for this service tend to be high and variable so it may not be appropriate for all tasks.
• Spread Queue ([[http://search.cpan.org/~jmay/Spread-Queue-0.4/Queue.pod|alpha software]]
• Starling
(apparently, twitter ditched starling for scala)
• Houston From Zivtech, not much documentation yet

Others mentionned here: ActiveMessaging, BackgroundJob, DelayedJob, and Kestrel.

Requirements

All the above must have:

• Open Source - It's free as in beer and speech

This section is currently being written, so its probably going to be full of errors, and possibly not that useful. Feel free to edit.

I'm probably going to pop a nice table of contents here, for now look to the right...

Contexts are simply a way for Aegir to store certain pieces of information that it thinks may be useful later. They are named so that they can be referred to easily.

Every server, platform and site in Aegir has an associated context. Each one of these contexts stores information about the associated entity. So, for example, a site context stores the base URI of the site. Later, the code responsible for informing the web server of the site's existence can look at this context and determine the requested URI.

In theory contexts allow most of the back-end commands in Aegir to operate just on the name of the context. This is an important concept: the alternative would be to try to guess everything that the command being invoked might want to know and pass it in as a series of command line arguments.

For example, the command that Aegir uses to clone a site just needs to be given the context of the site to clone and the platform to clone it to. From there the code can work out where the destination platform actually is, what servers it needs to talk to, and with what credentials.

Drush contexts

Note that Drush also has the concept of a context, but this is not to be confused with Aegir contexts which are different. Aegir contexts are closer to, and stored as, Drush 'aliases'.

Contexts are always named to start with a '@' symbol, except for in the filename of the file in which they are stored.

Contexts in Aegir are stored as a simple array of data in a file within the backend of Aegir. However this array of data is accessed and modified with a set of objects and accessors. Here is an example of what a context file looks like:

These files are stored in the /var/aegir/.drush directory on the master Aegir server.

In the above example we can see the properties of the context, like the 'uri' of the site, and the 'profile' that the site is running.

There are also a number of more interesting properties in the example, note the 'db_server' property, which has a value that is another context. We shall see later that these properties are special, and allow developers to easily access functions of the 'db_server' without needing to care which db server they are talking to.

Contexts are used within drush commands as objects, which subclass provision_context. This allows much more flexibility and cleverness, though can make them very confusing to use sometimes! As a developer you will only need to worry about the context objects, as Aegir handles storing them in the files for you. But it is important to note that each context must be representable in a flat text file (or be prepared to do some serious leg-work) by that I mean you probably don't want to be storing massive amounts of relational data in them, use a database for that!

Getting the current context is really easy, just call the 'd' accessor:

What the current context is will depend on the drush command that is running and the context that was passed into that drush command. For example if you ran a site verify task, then the caller would name a site context, and thus initially calls to d() would return the site context. The values of the context are accessible as simple properties of this object.

So, suppose a verify task is started for the main Aegir frontend, which has a context that is always called '@hostmaster', the drush command invocation would look like this:

Then the drush command for provision-verify could do this:

Which would print the type of the context passed to the verify command, in this case 'site'.

The d accessor is not to be feared! It is used all over the place in Aegir and if you're not sure what context you're going to get then you can always print d()->name and you'll get the name of the context that you're dealing with.

Loading a named context

You can pass an optional argument to the d accessor of the name of the context that you want. So, if you feel compelled to access a property about the main Aegir frontend site anywhere you can do this:

though most of the time you'll be dealing with the current context, and contexts that it references.

'Subcontexts'

Here's an example context:

Notice that some of the properties are the name of contexts, such as 'db_server' which has a value of '@server_localhost'.

Suppose now that I have some code that wants to get a property of the db_server associated with the @hostmaster context, I can simply do this:

The provisionContext object will return full context objects for properties that store the names of other contexts. The how and why of determining which to return as strings and which to return as a context object will be covered later. It is entirely possible to have the name of a context stored as a property in a context, that when accessed returns the string, rather than the context named in the string.

Services are the way in which new properties are added to contexts, in fact if you wish to indicate you want to store additional properties in a context, this must be done by a service. There is no other way to store data in a context. One might assume that you can just ask provision to save an additional value on a context, but this will not work, and you'll probably get very frustrated indeed!

For example, the provisionService_pdo service adds a 'master_db' property to the server context, it does this in a method thus:

This means that the 'master_db' property can now be set in a provison-save drush command, and will be persisted when the context is saved.

A service may also define properties on the context as actually representing another context, so that you may access that subcontext directly. You can see an example of using this subcontext in the 'Implementation' section of this Provision Contexts guide, but the an example of how you let provision know that a property name is not just a string, but a named context is with a method on the service:

Here the provisionService_http is being asked to 'subscribe' to a platform, it sets a property on the context, as above, but additionally uses the is_oid() method to indicate that the property is a named context. It then uses that immediately when it gets the name of the web_server context: $context->web_server->name. We'll worry about what 'subscribing' to a platform means later,.

This is stub page that will describe the way that you can use the service() method on a context.

We should add some documentation about Provision services, what they are and how they work and interact with contexts.

While working on the security enhancements, I stumbled upon a few fundamental UNIX limitations with groups. These limitations similarly apply to GNU/Linux.

1. Group name length limits

A group name (e.g. dialout) has a length limit, usually hardcoded in the operating system. This limit varies across different UNIX systems and implementations.

• NIS: 8 characters (source)
• HPUX: 8 characters, 16 starting from HPUX 10 (source, source)
• AIX: 8 characters, 255 for AIX 5.3 and above (source)
• Debian: 16 characters (source, defined in shadow at compile-time)
• Redhat: 16 characters, 32 starting from 2005 (source, source, source)
• Solaris: 8 characters? (source)
• Active Directory: 11 or 64 characters (source unclear)

In general, we can probably assume that 16 characters is a safe limit, and that's what's being used in hosting_client_sanitize.

2. Group membership limits

A UNIX user has a primary group, defined in the passwd database, but can also be a member of many other groups, defined in the groups database. UNIX has a hardcoded limit of 16 groups a user can be a member of (source). This means that if we translate Aegir clients into UNIX groups and you have access to multiple Aegir clients, you'll be able to access only 16 of those clients at a time. Rather annoying.

The problem is related with NFS, or rather AUTH_SYS (source), which has an in-kernel data structure where the groups a user has access to is hardcoded as an array of 16 identifiers (gids to be more precise, see this post for details). Even though Linux now supports 65536 groups, it is still not possible to operate on more than 16 from userland, through NFS - on a regular ext3 filesystem, it works without problems (tested in Debian Squeeze).

Solutions for this are not obvious. The blog post on sun.com proposes a technical enhancement to OpenSolaris. It's unclear whether that fix was implemented. This SAGE post explains that using GSS_API (used in NFSv4) should (should!) resolve the issue.

This more thorough analysis of the problem is more direct, to shamelessly quote it:

• Use NFSv4
• Use RPCSEC_GSS authentication
• Use ACLs

As mentionned above, this only applies to NFS mounts. NFSv4 should (should!) be simple enough and available on Linux. It is also unclear exactly how to use RPCSEC_GSS (is that Kerberos?). As for ACLs, we are fully supporting this now with the provision ACL extension, so that shouldn't be an issue.

N.B. The ACL-based approach, described below, has been implemented as an extension to Provision in http://drupal.org/project/provisionacl.

What's the problem here?

As things stand, you cannot run tasks as a regular UNIX user on your Aegir Drupal sites:

The reason the site is not found is because the alias cannot be loaded. If we change our home directory to find the alaises, we then get permission problems:

... so that doesn't seem to be a good approach, although one could just fix the permissions on the alias files for the site (which don't contain the DB password, oddly enough) and platforms (which have been safe since the "server" seperation):

Then we end up with another problem:

... and this is really the core of the issue here: we need to allow more than one user to access those files, and aegir must be able to fix those permissions when the site is created/verified.

So to sum up:

• a regular user doesn't have permissions to access the drushrc.php and settings.php files
• the aliases are not accessible unless you change your home directory (note that -i doesn't work either, for some reason)

Hackish solution: add user to more groups

Miguel indicates here that files should be writable by the Aegir group, and, if we start with this idea, we can actually get pretty far. The drushrc.php file is owned by the Aegir group, so adding yourself to that group and fixing up the file to be group-readable actually goes a long way. You also need to add the user to the www-data group since drush expects to be able to open the settings.php file also:

With the above changes, we can actually run drush commands!

We get a bunch of warnings, because provision can't load the server aliases (for a good reason - they contain DB passwords), but the command runs fine! In fact, if we want, we can avoid the luxury of using the aliases and just run the drush command in the site directory:

That warning can be removed by simply fixing the permissions on the drushrc.php file to be group-readable.

However, the problem with that approach is that the user has access to all sites, from there on. No regard to what client the user has access to. Obviously, this can work for a dedicated server (and it would need patches to provision so that drushrc.php files and optionnally aliases are group-readable) but it will not work for shared hosting, which is my prime objective here.

Patches necessary:

• group-readable drushrc.php (changes in provision)
• add users to the aegir and www-data groups manually
• optional: group-readable site aliases (so that users can access the aliases)

Issues:

• all or nothing: the user has access to all sites or none as we have a single group

Traditionnal UNIX permission-based approach

In order for this to work really correctly, we would need the following permissions setup. Instead of having the typical following:

(I am skipping libraries, themes and private directories for simplicity - libraries and themes are the same as modules and private is the same as files.)

... We would need to have the following:

That is, all files and directories are readable by the "client" group (which allow inter-client isolation) and some would be readable or writable by the webserver (which allows for uploads and the webserver to read the settings.php and so on).

In this scenario, the webserver has access to all sites all the time: it can read/write uploaded files of other modules, but, because of database cloaking, it can't access other sites' databases. If we want to fix this we would have to go with ACLs.

Note that this setup doesn't require the webserver running any special patch or daemon to run the PHP process as a normal user: it still runs as the www-data user.

Patches necessary:

• patches mentionned above (group-readable drushrc and aliases)
• creating the client group and adding users to it
• adding the aegir user to the client group
• patching provision so it changes the group of some files to the client group
• patching provision so it changes the owner of some files to the www-data user (this actually requires root: ouch!)

Issues:

• requires aegir to run as root
• aegir need to be a member of every group

The ACL approach

This is based on this excellent comment from malclocke.

This is one of the simplest and elegant approach, as it:

• doesn't require root access
• fits well with the n to n client access list model of aegir
• works without having to change the existing permissions (so would fit well in a contrib module)

To configure ACLs, you need to add the option to your mountpoint in fstab and remount the mountpoint (or reboot):

The rest is fairly simple: we just need to add access to the required groups to the drushrc and settings.php files:

... where "client" is the client group.

Optionnally, we can also give access to the alias files the same way, but we end up with the same warning on the server level files.

Note: this is a good introduction to ACLs, in Redhat

Patches necessary:

• make platform-level drushrc readable by all (or add an ACL for each client with access to the platform..)
• patch provision to add relevant ACLs on verify and install, to drushrc.php and settings.php:
• system-level configuration of mountpoints and extra package
• creating the client group and adding users to it

Issues:

• requires system-level modifications
• NFS has not supported ACLs very well traditionnally (see the NFSv4 implementation notes and the issues page for ACLs for more information) - but according to my tests (vanilla Debian squeeze), it just works

Other similar research

This is actually a well-discussed topic in the community, although nobody came up with a clean and complete solution for this.

• Brilliant comment from mig5 explaining group-ownership of key files
• A similar solution proposed here actually shows which chunks of code need to be modified in 0.4-alpha6
• Aegir permissions best practices thread - mentions everything from the "aegir group" to ACLs
• My original research into this problem

Aegir is designed to be easily extendable by developers. As it is made with Drupal and Drush, it is made of the hooks and command you know and love. If you are a user or admin looking to deploy contrib modules, you should look into the contrib modules list and user documentation instead.

What can I extend exactly?

These extensions may come in the form of

• Adding new tasks to be performed against sites
• Adding new services or implementations of service types (postgres for the DB service, for example)
• Overriding or hooking into existing forms such as the site form, to send extra data to the backend
• Using APIs to inject bits of configuration into configuration files such as settings.php and vhosts.

This area is be devoted to teaching you how to extend and develop for Aegir to encourage contributions to the Aegir project or to help you modify Aegir to suit your unique use case.

Aegir API documentation

The inline documentation is a good start to understand the various hooks and internals that allow you to extend and customize Aegir to your liking. The documentation is rendered on api.aegirproject.org daily.

See also the developer cheat sheet.

Please submit any suggestions or bug reports to the Aegir Project issue queue of your choice, under the "documentation" component.

Example modules

The contrib modules page documents all known Aegir extensions in existence. There are also specific developer documentation pages below.

Contrib developer documentation and roadmaps

Contrib developers are free to use this space to document the internals of their modules or their roadmaps. There is already this documentation:

• E-Commerce Integration Roadmap (uc_hosting)

1. Locations and paths

• Site URI: d()->uri; // ex: $base_url = 'http://' . d()->uri;
• Drupal root: d()->site // returns: /var/aegir/platform/drupal-6.20/
• Site path: d()->site_path // returns: /var/aegir/platform/drupal-6.20/sites/example.org/

2. File permissions and ownership

 $ctools_path = d()->site_path . '/files/ctools';

 provision_file()->chmod($ctools_path, 0770, TRUE)
   ->succeed('Changed permissions of @path to @perm')
   ->fail('Could not change permissions @path to @perm')
   ->status();

 provision_file()->chgrp($ctools_path, d('@server_master')->web_group, TRUE)
   ->succeed('Change group ownership of settings.php to @path')
   ->fail('Could not change group ownership of settings.php to @path')
   ->status();

3. Enabling/disabling drush/provision extensions from hostmaster

Every drush extension can implement a hook_drush_load(), and check if a module is enabled in the hostmaster site. See provision.api.php for an example.

4. References

http://api.aegirproject.org/

This article has been republished from mig5's website. Original URL

Aegir is a pretty powerful tool that allows you to very quickly provision new Drupal sites out of the box and manage them throughout their lifespan through a variety of tasks.

Its ability to understand different install profiles and thus 'distributions' shoots the awesome factor through the roof once you can deploy instances of OpenAtrium, Pressflow, ManagingNews or your own custom distro in just a couple of clicks.

Inevitably, though, your requirements will grow more complex. Aegir is about automation, and install profiles that prompt users for information through a series of tasks, and fail without that information, aren't much help. You might resort to a bunch of hacks to get that data into your client's site. You might try and insert some custom stuff into the site's vhost only to whimper in dismay when Aegir does a routine Verify sanity check and obliterates your changes.

If you get to this point, the simplicity of Aegir will be a distant memory and you'll be shaking your fist at us developers for forgetting about your corner case. Curses!

Rest assured, despite the constantly shifting development in Aegir and its early alpha stages, we try to maintain an ability to hook into Aegir and extend it as easily as possible, and it will only continue to get easier, especially once our API stabilises and is published.

Today I'm going to show you:

• how to add a custom Hosting submodule to Aegir
• how to alter the site form to add custom fields of data
• how to send that data to the backend
• how to apply that data to site vhosts and inside your custom install profile and site database

There are a few files here, because we're creating a custom module and install profile, and a few other things. For the impatient, I've attached a tarball to this article called 'cheesy.tar.gz' that contains all the relevant components and a README.txt instructing where to put it all.

Those of you who follow me on Twitter know I like to perpetuate the semi-French stereotype and rant and rave about cheese. I spent some time trying to think of a non-cheesy example to use in this tutorial, before realising what a silly idea that was. You can never have enough cheese :)

Part one - Adding a cheesy module

We'll add a custom module to the Hosting frontend called 'Cheese'.

Hosting is the 'frontend' bit of the Aegir system that allows you to plug data in via a web interface and have that information, if necessary, communicated to the backend Drush/Provision system by way of 'tasks'.

Our new module is simple: we're going to invoke a hook_form_alter() to add a textfield to the site form. The module will add a table to the Aegir database that stores this value, linked to the site nid. Keeping it out of the hosting_site table means we can disable and uninstall this module later and clean up properly.

There's nothing new or unusual about a Hosting submodule than any other Drupal module. Make the directory /var/aegir/hostmaster-0.x/profiles/hostmaster/modules/hosting/cheese

Info file

Here's our hosting_cheese.info

You can see we depend on hosting_site, because hosting_site is the module that provides the site form for installing sites!

Install file

Now let's add an .install file that defines our database table schema and how to install/uninstall it. Standard Drupal stuff.

The module

Here's our simple hosting_cheese.module file.

It's pretty straightforward: our first function is a hook_form_alter() of hosting_site_form(). We are adding a single textfield to the form called 'Cheese'. Because it's my favourite, the default value is going to be 'camembert' if it hasn't already been set on the node (if we're updating a site rather than creating a new one).

The next few functions are invocations of common database hooks such as insert, update and delete. They will insert the 'cheese' value from our form into the hosting_cheese table.

I call hook_nodeapi() to do $stuff on those actions. There's validation to check that a cheese was entered. We invoke hook_load(), which allows various modules and features to return their bits and pieces from various parts of the database as 'additions' to the core node attributes. With this we can call $node->cheese when we might need to.

At this point, you can go to /admin/build/modules and enable the Cheese feature. The database table will get installed, and you can now go to Create Content > Site and see our cheese field is present in the site form!

Part two - sending the cheese to the backend

Yes, it's very exciting that we have cheese in Aegir now. But don't provision a site yet! The fun hasn't even started.

In the same directory as the hosting_cheese module, create a new file called hosting_cheese.drush.inc. This is a Drush file that is going to be responsible for catching when an Install or Verify task is being executed against a site, and to send the cheese value along for the ride to the backend.

In this code, $task->ref represents the node object (the site).

Because of our additions in our implementation of hook_node_load(), the cheese value is fetched and available for us to use.

Part three - an install profile (just for fun)

I'm demonstrating all this to show you how to get data from the frontend (in the site form) to the backend. We've achieved that, but you probably want to actually *do* stuff with the data.

I'll demonstrate two things you can do with this data. They are just examples - Aegir isn't limited to this at all.

cheesy_sites.profile
First let's create a simple install profile called 'Cheesy sites'.

This is not meant as an exercise in learning how to write install profiles so I'll gloss over some of the details here. Basic version: we define our modules that the profile depends on (core + some contrib), we install those modules, and finally we do a basic variable_set to store a 'cheese' variable in the site's database, fetching the value from the Drush/Provision backend (sent by the frontend per above), defaulting to 'camembert' if there was none.

Obviously in a normal install profile, this file would be bigger, as you'd be doing a bunch of other stuff such as setting up node types, setting up menus, blocks, users etc.

cheesy_sites.make
I provide a Drush makefile here to fetch the contrib for you. This is totally out of the scope of the tutorial, but it's just fun to use Drush Make and I encourage you to get used to it :)

Download Drupal into /var/aegir/drupal-6.19 and copy the cheesy_sites directory into the profiles directory of drupal-6.19, or use a Drush stub makefile to do it all.

If you downloaded core manually, you'll want to cd drupal-6.19; drush make profiles/cheesy_sites/cheesy_sites.make --no-core. The contrib will be downloaded to /var/aegir/drupal-6.19/sites/all/modules

Add the platform

Add a new Platform in Aegir with the Publish Path being that which we just created: /var/aegir/drupal-6.19. A Verify task will be spawned, and Aegir will pick up all the modules and also the 'cheesy_sites' profile, which you can check by viewing the 'Installation profiles' section of the 'Packages' menu tab on the platform node, after the Verify task has completed.

You could also add this profile and its dependencies to an existing platform and just re-Verify the platform, and the new profile will be synced up and added to this platform's package registry.

Part four - make the backend knowledgeable about cheese

Now we've set up the frontend to get the cheese, set up a drush file to send the cheese to the backend, and we have an install profile that expects to find cheese and set a variable about it in a database when installing a site.

We've left out one crucial stage, and that is to actually prepare the backend to tell it that cheese is coming!

cheese.drush.inc

Telling the backend about the incoming cheese is very similar to what we did earlier about capturing the cheese info on specific tasks and do $stuff with it.

To tell Drush and Provision (Aegir's Drush extension) about the cheese, create a file in ~aegir/.drush called cheese.drush.inc.

We'll invoke a Drush hook that sets the value for the backend on installing a site.

This saves the value into the site context, where it's usable by the install profile. This occurs in the pre hook, prior to when Aegir starts executing the tasks outlined by the install profile.

Part five - using Aegir hooks to inject settings into the site vhost

In Part four, we'll have accomplished what is needed to set the cheese value for the backend, which in turn means you can now create a site in the Aegir frontend, choose the 'Cheesy site' profile, and expect the install profile to pick that cheese setting up and store it in the new site's database as a variable.

But! While I'm here, let me show you another Aegir hook that allows you to safely inject extra configuration sent from the frontend into the site's Apache vhost config file, persistently, without it being overwritten next time the site is Verified.

We'll invoke an Aegir hook called provision_apache_vhost_config() to insert a SetEnv into the vhost file, just for the hell of it.

Another example where you might want to do this is, say, to set a htpasswd password in the site form, and use the hook to inject mod_auth htpasswd protection for the site (perhaps a demonstration for another time!).

Add this to your cheese.drush.inc:

The return value is inserted into the vhost. You can return multiple lines by wrapping them in an array, or even more elegantly, simply return a path to a template .tpl.php file and do your customisations in the template!

Part six - check the results!

If you've done everything up to Part four and five prior to installing a site using the 'Cheesy site' install profile, do so now!

I'm going to create a site called 'cheesy.mig5-forge.net' and set my cheese type to be 'cheddar'.

Now let's check to see those settings in their end result:

The variable in the new site's database

Run this command: drush @yoursite.com vget cheese

You should see something like this:

Great! Now let's look at the Apache vhost file for this site, and we should see a 'SetEnv' parameter injected into the vhost.

Conclusion

I know these have been silly examples, but I hope you've taken some important lessons out of all this:

• How to extend the frontend of Aegir and hook into things like the site form and site node
• How to capture data on task 'events' like install and send it to the backend
• How to recognise that data in the backend and use it in install profiles or invoke our hooks to inject the data into configuration files
• How much I really like cheese

Cheesy tarball

As mentioned at the start of this article, you can download all these components I've been talking about here: cheesy.tar.gz. Remember to read the README.txt

Further reading

A great article by hadsie on g.d.o on sending data to an install profile via the Signup form covers similar information on capturing data sent to the backend in an install profile.

Steven Jones has published a HTTP mod_auth feature extension for injecting htpasswd password protection on sites over at https://github.com/computerminds/aegir_http_basic

Related links:

• For installation and configuration instructions, see the Administrator Manual.
• There was some valuable discussion on use cases on groups.drupal.org.

Don't put feature requests in here. Submit them to the uc_hosting or hostmaster issue queues instead.

1. Drupal 6 / uc_hosting 1.0

Aiming for compatibility with:

• Hostmaster 1.0
• Ubercart 2.4

Structure:

• uc_hosting: This implements shared functions and classes, including the function to create a client based on an ubercart order and product.
• uc_hosting_products: This is where we put the ubercart function calls, attribute generation, etc... This module is intended to be used alone for the "my ubercart is on my aegir site" scenario.

1.1. Goals

1.1.1. Single site products

In.

1.1.2. Multi site packages

In.

1.1.3. Recurring payments

We need to test the module using uc_recurring.

1.2. Nice-to-haves

Given where we are at now, this stuff will most likely not make it in until an eventual 1.1 release. The exception being Hadsie's work on "try before you buy".

1.2.1. Purchasing wizard

Stills needs to be developed. The create my site now option is a start and is already in.

1.2.2. Make it unnecessary to log single-site purchasers into the aegir site

Needs some testing but should be doable. Maybe just needs a little documentation. I believe Hadsie's work is related to this.

1.2.3. Deferred payments

Ergonlogic is working on this.

2. Drupal 7 / uc_hosting 2.0

We are still unsure about whether to support commerce, ubercart, or both with this module. We will most likely wait until development on the D7 port of aegir starts before beginning this work so we will see then what the landscape looks like.

Aiming for compatibility with:

• Hostmaster 2.x
• Ubercart 3.x, or Commerce 1.x (maybe both?)

2.1. Goals

2.1.1. Remote storefronts

Hopefully, this will be supported by plans for new xmlrpc methods in the hostmaster signup module.

2.2. Nice-to-haves

2.2.1. Desjardins recurring payments

This is a seperate module being developed by Koumbit.

2.2.2. Importable demo products

This would be really great.

2.2.3. Simple procedure to deny user 1 to certain clients

Ergonlogic has published a module that should make this possible: http://drupal.org/project/hosting_profile_roles.

3. Wishlist:

• A decent ubercart import function (see this and this... this and maybe this)

4. Unanswered Questions

1. Will the Drupal 7 version use Ubercart or Commerce? Both? Neither?
2. How and when will people be billed? And what degree of control will uc_hosting offer over that?
3. Aegir has clients & UC has clients. How are they related/distinct? How do we keep this clear to users/admins?

This document assumes you want to implement a product feature using uc_hosting's database tables and exiting infrastructure. For a more complete example of an Ubercart 2.x product feature implementation, check out uc_file in the Ubercart core.

The Basics

Any module implementing a uc_hosting feature should include both uc_hosting and uc_hosting_products as dependencies. Include these lines in your .info file:

dependencies[] = uc_hosting
dependencies[] = uc_hosting_products

Declaring product features to Ubercart 2.x

The first step is to implement Ubercart's hook_product_feature. Here is an example implementation for the platform access feature in uc_hosting_products:

/**
 * Implementation of hook_product_feature().
 */
function uc_hosting_products_product_feature () {
  $features = array();

  // Set the feature for the platforms
  $features[] = array(
    'id' => 'hosting_platform',
    'title' => t('Access to a platform'),
    'callback' => 'uc_hosting_products_platform_form',
    'delete' => 'uc_hosting_products_feature_delete',
    'settings' => 'uc_hosting_products_platform_settings',
  );

  return $features;
}

An implementation of hook_product_feature returns a numerical array of associative arrays. Feature arrays must include five keys:

• id will be used to identify your feature at the database level, in the uc_product_features table, and also in urls. Think of it as the type of your feature.
• title is the name of your feature as it will be displayed to site admins.
• callback will be used to create and modify instances of your feature attached to specific products. This callback should return a drupal form passed through the uc_product_feature_form function.
• delete is a simple function called on the removal of a feature instance from a product.
• settings is a callback that provides additional settings for the feature, and is not used presently in any uc_hosting product feature.

Of note is that, unlike Drupal's core hook_menu and hook_theme, hook_product_feature does not include a 'file' key. uc_hosting works around this by using include_once at the beginning of the hook_product_feature implementation.

If you wish to use uc_hosting's shared feature functions, make sure to include the file "products/inc/uc_hosting_products.feature_shared.inc" (path relative to the uc_hosting root) at the beginning of your implementation of hook_product_feature.

The callback form

This form is called everytime someone adds a product feature to a product. It allows you to set any options for your feature that should be delivered on a per-product basis. Lets take a look at this function for the platform access feature:

/**
 * Callback to add the platform form on the product feature page
 */
function uc_hosting_products_platform_form ($form_state, $node, $feature) {
  $form = array();

  // Get default values and shared form elements for all uc_hosting features
  $hosting_product = _uc_hosting_products_feature_fetch_product($feature);
  if (empty($feature)) {
    $feature = array('id' => 'hosting_platform');
  }

_uc_hosting_products_feature_fetch_product attempts to retrieve existing values for this instance of a product feature from the database.

  _uc_hosting_products_feature_shared_elements(&$form, $node, $feature, $hosting_product);

_uc_hosting_products_feature_shared_elements declares some form elements that either uc_hosting or Ubercart itself depend on to provide the product feature functionality.

  // @see _hosting_get_platforms()
  if (user_access('view locked platforms')) {
    $platforms = _hosting_get_platforms();
  }
  else if (user_access('view platform')) {
    $platforms = _hosting_get_enabled_platforms();
  }
  else {
    $platforms = array();
    drupal_set_message('You must have permission to access platforms to enable this feature.', 'warning');
  }

  $form['platform'] = array(
    '#type' => 'select',
    '#title' => t('Platform'),
    '#description' => t('Select the platform to associate with this product.'),
    '#default_value' => $hosting_product->value,
    '#options' => $platforms,
    '#required' => TRUE,
  );

This code block generates a list of platforms to allow the admin to select which platform to bind to his product. This code is specific to the platform access implementation - this is where you should include your own form elements relevant to your feature.

  $form['#validate'][] = 'uc_hosting_products_single_feature_validate';

For many implementations, uc_hosting_products_single_feature_validate ensures that a given uc_hosting product feature can only be enabled once on any given product. All of the exising uc_hosting_products features make use of this function.

  return uc_product_feature_form ($form);
}

Finally, uc_product_feature_form adds some required form elements from Ubercart.

You also need to make sure to code a submit function for your callback, of the form callback_submit:

/**
 * Save the platform feature settings.
 */
function uc_hosting_products_platform_form_submit($form, &$form_state) {
  $hosting_product = array(
    'pfid' => $form_state['values']['pfid'],
    'model' => $form_state['values']['model'],
    'type' => 'platform',
    'value' => $form_state['values']['platform'],
  );

  $platform_node = node_load($hosting_product['value']);
  $description = t('<strong>SKU:</strong> !sku<br />', array('!sku' => empty($hosting_product['model']) ? 'Any' : $hosting_product['model']));
  $description .= t('<strong>Platform:</strong> !platform', array('!platform' => $platform_node->title));

  $data = array(
    'pfid' => $form_state['values']['pfid'],
    'nid' => $form_state['values']['nid'],
    'fid' => 'hosting_platform',
    'description' => $description,
  );

  $form_state['redirect'] = uc_product_feature_save($data);

We manually save the product_feature to ubercart's database tables here. This is so that we can then retrieve the index of the entry in the uc_product_features table for later use in our own data.

  // Insert or update uc_hosting_products table
  if (empty($hosting_product['pfid'])) {
    $hosting_product['pfid'] = db_last_insert_id('uc_product_features', 'pfid');
  }

  $existing_prod = db_fetch_object(db_query("SELECT hpid, data FROM {uc_hosting_products} WHERE pfid = %d LIMIT 1", $hosting_product['pfid']));

  $key = NULL;
  if ($existing_prod->hpid) {
    $key = 'hpid';
    $hosting_product['hpid'] = $existing_prod->hpid;
    $hosting_product['data'] = unserialize($existing_prod->data);
    $hosting_product['data']['platform'] = $hosting_product['value'];
  }
  // If necessary build a data array from scratch
  else {
    $hosting_product['data'] = array(
      'platform' => $hosting_product['value'],
    );
  }

  $hosting_product['data'] = serialize($hosting_product['data']);

  drupal_write_record('uc_hosting_products', $hosting_product, $key);
}

Note that the uc_hosting_products table has both an integer column value, and a longtext column data for storing serialized information about the feature.

Taking action in Aegir based on product features

uc_hosting assumes that most purchases made via Ubercart are intended to be expressed as changes to an Aegir client node. So it provides some code to help you create new clients or modify exiting ones when an order containing your feature is made. Any other actions needed to make your feature work you will need to implement yourself. This will involve implementing the Ubercart hook, hook_order and your own callback function.

Lets start by taking a look at uc_hosting_products own implementation of hook_order:

/**
 * Implementation of hook_order
 *
 * Actions t$form_state['values']['create_later']o take on order changes involving an aegir product
 *
 * @param $op string
 *   Provided by ubercart on invocation
 * @param &$arg1
 *   Different data depending on the op
 * @param $arg2
 *   Different data depending on the op
 */
function uc_hosting_products_order ($op, &$arg1, $arg2) {
  switch ($op) {
    case 'update':
      if ($arg2 == 'completed') {
        foreach ($arg1->products as $product) {
          if (_uc_hosting_products_has_feature ($product)) {
            // Make the changes necessary to the hosting client
            $client = uc_hosting_update_client($arg1, $product, 'uc_hosting_products_client_update');
          }
        }
      }
      break;
    default:
      break;
  }
}

_uc_hosting_products_has_feature is defined in uc_hosting_products.module, and provides some simple logic to detect uc_hosting product features. Rather than use this function, you will most likely want to define your own conditions.

uc_hosting_update_client does the work of matching up an incoming ubercart order to an existing Aegir client, if possible. If not possible, it creates a new client node. It then calls the function provided as a third argument, in this case uc_hosting_products_client_update, and passes to that function the affected client node, as well as the product and order objects from Ubercart.

This callback function is the place to take actions in Aegir or pass commands. Take a look at uc_hosting_products for a fully fleshed out example.

Provision has been refactored in 6.x-2.x to use class auto-loading. This simplifies use of classes, as it obviates the need to explicitly include the files in which the class and it's parent classes are defined. In addition, it standardizes the placement of class definitions within a hierarchical file-system structure and will make name-spacing simpler if/when we begin to adopt Symfony components.

While not strictly required in contributed modules, it is highly encouraged. If you maintain a contributed Provision extension, some re-factoring will be required to support class auto-loading.

First off, you'll need to add a registration function that will enable Provision to autoload your extensions classes. Here's an example from provision_civicrm:

Next, your extension will need to call your new function in a hook_drush_init(), so that it is run as early as possible during a Drush bootstrap.

Finally, you can move your class definitions into the proper file-system structure. In provision_civicrm, we defined a new (stub) service to allow saving data from the front-end into a site context (entity/alias). So, we created a file at Provision/Service/civicrm.php that included the class definition:

Non-standard Aegir installs may require adjustments to the default configurations for a platform that are defined by Aegir during the creation of a platform. These templates can be found in /home/aegir/.drush/provision.

Editing the actual templates prevents Aegir from overwriting your customizations when the verify task is run on an live platform.

Below are examples of customizations made to non-standard Aegir installs

Running an Aegir platform on a unix socket for php-fpm on Nginx

First let's find out where provision has the default config templates for our platfom.

We can see that the config templates for Nginx direct all php requests to port 9000 on the localhost I.P. Let's change that to a unix socket (which avoids the slight delay associated with tcp connections). After commenting out those lines and adding our custom lines which will use a unix socket instead of a tcp port, we get..

Remember to do the same to your platform config files in /home/aegir/config/includes. Next time you verify your platform your customizations will remain after verification.

It's possible rely to on manipulating the drush context for certain tasks and using hook_post_command to make use of that data. In this case, I wanted to save values to the site's drushrc.php file.

Add the following code to hook_post_provision_verify:

This makes sure that my_custom_option perpetuates through these potentially destructive operations. Since it is provision-verify that writes the drushrc.php file for the site, drush options set in the site context during hook_post_provision_verify will be saved.

The final step is to make sure the verify hook is run with the appropriate options at the end of my migrate task, so add this to hook_post_provision_migrate.

I got this idea from looking at the drush_provision_drupal_provision_migrate function in platform/migrate.provision.inc.

I used this approach to create a patch for provision_civicrm

This was originally documented in the provision issue queue.

This section of the documentation deals with integrating Aegir components with other services (as opposed to extending or overriding direct Aegir functionality itself).

Examples might include using Aegir's database to authenticate users based on their presence as 'clients' in the Aegir system, e.g authenticating FTP access to sites for clients that have those sites managed in Aegir.

Adam has written an article that explains how to hook up OpenSSH and PAM on your Linux server, so that you can authenticate your clients via the Aegir database and have them SFTP/SSH into your server and be chrooted (locked) into just seeing their sites (or a platform) with a restricted command(s) shell.

Read the article here.

Cafuego has written an excellent article that explains how to hook up PureFTPd and PAM on your Linux server, so that you can authenticate your clients via the Aegir database and have them FTP into your server and be chrooted (locked) into just seeing their sites.

Read the article here.

Testing server URL: http://ci.aegirproject.org

Recently, mig5 put together a testing server based on Jenkins.

So what? In doing so, he's produced a tool that will allow us to spin up a temporary virtual server, drag in our codebase from Drush Make, other codebases as platforms, and do test installs of sites on it. This should become a routine start-to-finish test of our applications, features, modules, etc. All at the click of a button (or less, by scheduling it!)

Currently, we have a build project that:

• Provisions a new VPS at Rackspace Cloud
• Installs Aegir 6.x-1.x branch
• Builds a Drupal 6, Drupal 7 and OpenAtrium platform using Drush Make
• Installs a site (with respective install profiles) on all three platforms

If anything fails here, an alert will come through to #aegir on Freenode. This will help us to test the Aegir install (especially to ensure our compatibility with Drush and Drush Make releases), and to identify any regressions in our own project.

Every 15 minutes, Jenkins will check if there's a new commit to hostmaster and provision, and if there is, fire up a new install and test a few things to find regressions.

We also have a build project for the most recent official release of Aegir (@TODO: test .deb installation method of Aegir for this), as well as a build that installs an earlier Aegir and then upgrades to the latest 6.x-1.x to test the upgrade path.

Also, steven has created a sandbox to host the code that runs the tests on the jenkins server.

We have seen numerous waves of spam on this site. At first we setup mollom, in 2011 but sometimes it is not enough.

Current practices

The very first step is to identify the user account and disable it (not delete it, we need it to find the content to remove).

Views Bulk Operations views used for despamming:

• comments
• nodes

Note that this will flood your screen with drupal_set_message for every page that is deleted, which may make the site difficult to use for a while. To top that, the final page will list the name of every page that was removed, just to make sure you had enough spam for the day.

You can also use the regular comment view to delete content and report it to mollom if you are patient enough.

You can also delete the account if you are motivated and want to grind that specific axe. Look at the regular user listing for the latest user accounts created.

Other ideas

• http:BL - used on Drupal.org
• blogspam - used on debian-administration.org, but not ported to new API
• spambot - similar
• rate - may be used to process small amounts of spam by the community
• botcha