Following the Drupalcon Chicago 2011 BOFs discussion, the community came up with a series of ideas and wishlist items for the upcoming 2.0 release. This roadmap may be updated or trimmed down as development goes ahead.

We do not commit to doing all of this for 2.0, this is mostly a wishlist of what people expressed were priorities for them during the brainstorm. If people want things to move forward, they will need to submit patches or create feature branches, basically lead development on the feature you want. This page can be used to coordinate overall leadership of those issues, while pointing to the Drupal.org issue queues for the specifics.

This roadmap supersedes the previous 1.0 roadmap.

Update: this roadmap is very ambitious, and should be considered more as a "wishlist". What we agreed on during the february 6th 2012 scrum was that we would focus on those items, in order of higher to lower priority:

1. Autoloaded classes (means we drop support for PHP 5.2 and below)
2. Drush 5 support
3. Drupal 7 port of the frontend

Once all those are done, 2.x should go in beta and once there are no criticals, it gets release, even if we don't have all the other features listed below.

1. Port frontend to Drupal 7

We were originally planning to release 1.0 with Drupal 7 support to make sure it was a "long term support" release, but this has changed - we hope to keep the release cycle short and therefore Drupal 7 was pushed back to 2.0. But it is still a target to make sure we a reliable release that will be supported for a long time.

Drupal 7 also introduces the concept of entities which could be very useful to lighten our many content types in the frontend. It could also be useful for DNS support as zonefiles could be modeled directly as entities.

This concerns of course only the port of the frontend (hostmaster.profile, eldir.theme and hosting.module), the backend already supports d7 installs and upgrades. The issue to followup is #1261030.

2. Drush collaboration

We want to collaborate more with the good people in the Drush project. We had good conversations at Drupalcon and hope to keep on having more about how to do staging, sharing code, working with contexts and aliases and so on.

This collaboration should permeate the whole 2.0 release cycle, as a lot of goals here overlap work already happening in the Drush project. We have a long tradition of leading development of new features in Drush, by first implementing them in Provision, and we should continue that trend, with a focus on merging those changes back up into Drush, and eventually removing them from Provision in 2.0.

Koumbit plans to keep on taking care of the Drush Debian packages and coordinate with the Drush team as much as possible.

2.1. Sending more code upstream

Our "contexts" code could be refactored and sent upstream. provision-save would be a likely first target, along with the persistent "contexts" classes. Note that we have hit a significant wall in the current API because it is hard to interoperate with existing server and site contexts from the outside of the class, so that's something that should be cleared up before going ahead with those merges.

2.2. Removing common code

At the very least, we should start removing common code between site-install and provision-install. The new drush archive command should also replace our builtin provision-backup, and provision-save may be refactored around drush_save_config() too (see issue #736686 for this).

2.3. Standard archive support

... which brings us to the standard archive format. Drush and Aegir developers have collaborated to create a common format to export and import Drupal sites. A single archive could bundle one or many sites, including site-wide code, so this is a significant difference (and improvement) over the current de-facto Aegir standard.

We want to support that new format. Full multi-site archives should be imported as platforms, while Aegir could keep on exporting single-site archives for standard platforms, like OpenAtrium or other distributions.

2.4. Unit testing

Drush has started to implement unit testing with PHPUnit in Drush 5 (see the tests directory for samples), and we should jump in that party, extending those classes and unit-testing the hell out of Aegir itself. That will make merging features into Drush easier and make sure we have no regressions while doing so.

3. Multi-server improvements

While Aegir 1.0 has multi-server support, it is far from ideal or complete, and we wish to harden that design in 2.0.

3.1. From a hub-spoke to a mesh model

The current model where there is one main server (the "hub") where all platforms and sites are created and then synced to remote servers (the "spokes") seems to be showing scalability issues and concerns about where the authoritative data should be.

For example, it's currently not recommended to allow developpers to access sites on remote servers, as the authoritative data resides on the hostmaster site. Another issue is that aliases are maintained only on the master server, and therefore are not accessible on remote servers. Finally, the fact that drush and provision are not installed on remote servers make some operations unnecessarily slow as we need to make the operation locally and then sync the result.

The alternative has yet to be discussed. We especially want to chime in with the Drush community here to see how the best practices should be. The conversation has started, and with the migration of Drupal.org to git, a lot of possibilities just opened up, with a lot of people being all of a sudden more familiar with this excellent too.

In general, it should be possible to expect remote servers to have drush and provision installed and be relatively independent from the "master" server. A design like this could eventually mean that Aegir could manage other Aegir sites and upgrade them. It is the "mesh" design, of which the "hub and spoke" model is only a subset.

This follows the practice of "tools not policy" we are heading towards: we leave the mesh vs hub/spoke decision to the site administrators and provide the tools to support both approaches the best we can.

Koumbit plans to spearhead this effort this summer.

3.2. Improve cluster support

The current multi-server model doesn't support clusters and shared filesystems very well. Aegir 2.0 should allow scaling your web hosts horizontally by spreading a single site amongst multiple servers. While 1.0 can setup sites on remote servers, it follows more the concept of "sharding", spreading different sites amongst different servers than load balancing all sites amongst all servers.

This involves supporting load balancers (e.g. IPVs, Pound, Varnish), maybe through the existing "cluster" service, but also shared filesystems (e.g. NFS, OCFS, GFS, GlusterFS), maybe through a new "fileserver" service.

Koumbit plans to spearhead this effort this summer.

3.3. Optimizing clones and migrates

We quickly mentionned git support above, but this could lead to significant improvements, for example by using git clone --shared instead of the traditional "backup and deploy" approach to clone/migrate sites.

There is certainly the possibility of rewriting clone to optimize away "backup and deploy" and merge them in a single operation that could use copy on write mechanisms like hardlinks (cp -al), git tricks (git clone --shared) or filesystem snapshots (e.g. BTRFS snapshot directory/file support).

Once deploy is optimized, we could consider hooking it the Drupal 7 or third party unit tests to rollback on such failures.

Those optimizations would need to integrate with and support the multi-server model elegantly too, especially the hypothetical fileserver service.

4. Modularization of more systems

Aegir 1.0 is already pretty modular: "services" are an abstract concept that can be implemented and inherited. For example, the "HTTP service" has four implementations: Apache, Apache-SSL, Nginx and Nginx-SSL. The database and DNS layers are also abstracted that way.

But more can be done here.

4.1. Modular backup system

Right now backups can be pretty heavy and use the venerable (1979!) tar backup format and utilities, which made Aegir portable to almost any UNIX operating system. It's a good start, but it's highly inefficient. As mentionned above for the clone and migrate optimization, we could also use modern filesystem features like snapshots or use git support. Another suggestion was to use S3 or similar services for offsite backups.

The basic idea here is that backups should be extendable - if you want to backup in your shoebox, Aegir should allow you to write a plugin for it. There was some initial work done by computerminds on github.

4.2. Modular platform support

This is also known as the alien platform support, that is - being able to install other "things" than Drupal in the backend, for example (oh the heresy!) Wordpress. This involves ripping apart the "platform" code in the frontend and the backend to make it more CMS-agnostic, so that Drupal is just one of the implementations.

This would make sense of course if at least another implementation would be supported. This could also involve merging the site and platform creation forms for those platforms that do not support multi-site like Drupal does (so that creating a site effectively create a platform).

4.3. Modular queuing system

The current queuing system is what we informally call the "ghetto queue". It's a cron job that pulls tasks from a MySQL database. While this scales well for a few sites, on busy servers it's a total nightmare, especially for mass migrations. Tasks stack up and concurrency is not managed at all. This needs to be completely redone.

A possibility is to use Drupal 7's abstracted queue API and let administrators implement their own queuing system according to their local APIs.

The new queuing system could be a key part in how the multi-server "mesh" model would be implemented. It could also be how non-Drush backends could be handled by Aegir - in that scenario, Aegir would become an interface for a standard queuing system, yet to be determined, although a lot of ideas are flying around: Jenkins/Hudson, AMPQ, Drush itself, etc. See this evaluation for that decision.

Koumbit plans to spearhead this effort this summer.

5. Other features

5.1. Automatic upgrades

There were a few discussions around managing platforms automatically. The implementation of hosting-import is one of the building blocks of this, but there remains some fine-tuning for this to work out of the box, especially for new users, some of which have requested that we can build makefiles straight from the frontend, a bit like how it's done on drushmake.me. One could also use feature servers feeds to update their platforms. Again, we should leave the policy to the user or at least make this pluggable.

This mostly consists of automating the creation of new platforms, the upgrade of sites between platforms and garbage collection of old platforms.

5.2. PostgreSQL backend support

The backend database layer (the "db class") now supports only MySQL, that is, while the Aegir frontend could technically run in PostgreSQL (although that is still unsure), the backend can't create sites in a PostgreSQL database.

Most of the work here will be around supporting user credentials handling, creation and destruction, which are fairly different between postgres and MySQL. The issue for this is #585796.

5.3. Statistics improvements

Work has already started at Koumbit to implement a "statistics" queue that would mostly pull Apache logfiles and parse it for size (and CPU timing) information. This is to implement bandwidth and CPU quotas, but could be extended in a more generic "statistics" queue that would collect all sorts of information:

• if your Drupals need to be upgraded or are secure (the basic "status report")
• how many modules/themes/profiles/libraries are installed (populating the frontend data)
• how many users/nodes/fields instances do you have?

All this could be collected in a nice dashboard that would outline client, site or platform-specific information.

5.4. Full DNS support

The DNS support in the backend is already pretty mature, but it's not really intuitive how it works and how to integrate it into existing infrastructures. Issue #922280 details all the next steps required here, but we can already say we hope to have a zonefile editor in the frontend, probably based on D7 entities, reasonable access control for zones, registrar support and "host my mail on gmail button" type of wizards.

This could potentially be picked up by a GSoC student.

5.5. Improved intersite security

We are already going through some hoops to make sure the database credentials are not accessible from one site to the other. The credentials are stored in the Apache environment and cleared out as soon as they are used, which makes finding them in settings.php impossible. Koumbit has also worked on a Provision ACL module that implements POSIX ACLs on files which allow more granular access controls on files.

But there are other security issues remaining that we should be looking into, especially the way Drush and Drupal are bootstrapped during tasks, which is currently insecure.

Koumbit plans to spearhead this effort this summer.

6. Contrib modules

Other contrib modules may have different roadmaps, but they must respect the release naming conventions. Contrib modules should aim for a 1.y release now, that will be compatible with the 1.x series. 2.x modules should be ported to Drupal 7 and follow the API (in development) of the 2.x series. Modules do not have to follow the minor version numbers of Aegir. For example, Provision ACL 1.1 is compatible with any release of the 1.x core series.

We are also considering making point releases of the backend and the frontend independent, so that Provision 1.1 and Hostmaster 1.2 could be release independently.

6.1. Other modules roadmaps

This is a short index of other modules roadmaps to 2.0:

• Ubercart integration (uc_hosting 2.0)