Clone a repo

Let's take hostmaster:

If successful, you will have a hostmaster directory at the last state of development ('HEAD' in CVS speak, usually 'master' branch in git speak, though in our case this is now 6.x-1.x).

Check the status of the local git repository:

See the commit messages:

Check all 'releases'

Create a new branch for your local development:

Create a new branch for development, based on an existing remote branch or tag

So say you have some work you want to do on the hostmaster module. You first need to clone the repository:

That will create a hostmaster directory with the whole history of the project, and a checkout of the "master" branch (which is the equivalent of CVS's "HEAD" in git).

You make some modifications to the code to fix a bug, and then, you make a commit. That commit will only be local for now, that's the way git works:

Note that contrarily to subversion or CVS, you need to explicitly mention which file(s) you want to commit. You can also use the -a flag to commit all changes.

If you have write access to the repository, you can now push your changes straight into git. However, the git:// url above is readonly, so you will want to use SSH to push your changes. For that, we will have had to add your SSH public key to our repositories. You may also want to add a special "remote" repository to push your changes. You will need to do this only once (per repository):

You can now push your changes to that repository.

If you do not have write access to the repository, you can send the Aegir developers patches by submitting them for review in the Issue queues (see the patch contributor guide on drupal.org for more details) or by uploading them to your personnal sandbox.

Aegir development makes a heavy use of branches to keep development stable. We use feature branches to do large development work and have release branches to keep stable releases maintained while development happen on the development branches.

1. Summary

• The current development branch is: 7.x-3.x
• The current stable branch is: 6.x-2.x

2. Branch naming conventions

All branches are either development branches, stable branches or feature branches. Development and stable branches have the same naming convention: 6.x-2.x or 7.x-3.x. The only different between the two is that a stable branch is the branch where stable releases are published from.

Note: since the great git.drupal.org migration, we are back on Drupal.org and have been forced to change some of our naming convention, especially to include the core release number in the tag (so 6.x-2.x), which is a source of confusion for our provision module which supports any Drupal core version. See this issue on drupal.org for followup.

2.1. Main development branch

The main development branch is created after a fully stable release is published. For example, the 6.x-2.x branch has been created when the 1.0 release was published. It is a development branch until it is considered feature-complete and goes through the alpha/beta/rc release cycle, at which point it will be declared a stable branch and a new development branch will be created (7.x-3.x at that point).

The development branch is not stable and should not be used in production environments.

Note that we have dropped the master branch as it is not well supported on Drupal.org for releases.

2.2. Stable release branches

Stable releases are performed from stable release branches. A branch is declared stable only after it went through the complete alpha/beta/rc release cycle, as detailed above..

Once the branch is declared stable, new releases are quickly done as "hotfixes", without intervening alpha/beta/rc releases. So for example, 6.x-1.0 will come after 6.x-1.0-alpha1, 6.x-1.0-beta1, 6.x-1.0-rc1 at least, and then be followed directly by 6.x-1.1.

Releases on the same stable branch are API-compatible with each other (so 1.1 is compatible with 1.0). The upgrade path is supported between major stable releases (so you can upgrade from 1.0 to 1.1 or 2.0).

Special case: during the 0.x release cycle, releases were not API-compatible with each other, ie. 0.4 was not compatible with 0.3, and you could barely upgrade from 0.3 to 0.4.

Only one stable branch is maintained at a time.

2.3. Feature branches

More complex development goes into development or "feature" branches, to avoid making master too unstable. Feature branches must be prefixed with "dev-" and use the issue number (if available or relevant) and a short name. So for example, there could be work on a dev-12345-mybug branch or dev-dns branch for a larger feature (DNS, in that case) which doesn't fit in a single issue.

Feature branches should be based of a known stable baseline as much as possible. For example, you should branch off the last alpha release if you want to make merging with other dev branches and master easier. See this article for a thorough discussion on this practice

3. Tag naming convention

Every release is tagged with a 6.x-x.y tag (e.g. 6.x-0.4, 6.x-0.4-alpha15 or 6.x-1.0). Those tags are usually laid on the stable branches, except for alpha/beta/rc releases which are laid on the development branch.

4. CVS and git history

The CVS tags and branches are still accessible in git and will be kept there for posterity. They respect the traditional Drupal.org naming convention.

They have also been converted to the new git.drupal.org conventions during the great git.drupal.org migration.

Since that migration (february 2011), history has been rewritten to clean up the authors in the history.

After thorough discussions where we weighted the pros and cons of profiting from the Great Git Migration, I think it's pretty clear that we have a great opportunity to migrate our source repositories back to Drupal.org. We will get:

• more visibility - people will find us through drupal.org easily
• more consistency - versions in the issue queue matching the real releases
• even more consistency - people will find the source better from drupal.org
• and then more consistency - usage stats will work again!

We will be able to:

• host our module and theme within the install profile - the major reason why we migrated away back then
• use whatever naming convention we want for tags and branches - although only some will be used for releases for now, see this issue for follow-up
• keep our history - the git team was nice enough to clone directly from our repositories (although see below, we're taking the opportunity of the migration to clean up our history)

What it means for you

This does have an impact on your existing repositories. Since we migrated to git.drupal.org, the repositories have changed location. The other thing is that the commit IDs have changed, and you will need to clone new checkouts before you can work with them effectively.

So in short:

• repos will move to git.drupal.org
• you need to clone again
• make sure your identity is set correctly

URL changes

So the repositories have moved. Everyone has to fetch the source code from git.drupal.org now. At some point in the future, Koumbit will close git.aegirproject.org. Concretely, Koumbit will keep the mirror running readonly for a while just to be on the safe side, but you shouldn't expect git.aegirproject.org to work and should switch URLs, as detailed below.

This documentation is taken from the Drupal.org git workflows documentation.

Readonly repositories URL change

The readonly access will change from:

to:

See also the git patch maintainer guide.

Core committers repositories URL change

The read/write access will change from:

to:

... notice how you will need to set your Drupal.org username in the URL. Drupal.org has good documentation on how authentication works on git.drupal.org. See also the project maintainer git guide.

Sandboxes

The migration to Drupal.org also introduces sandboxes, something we didn't have before, and which allows you to host your own branches of our modules (even if you're not a core committer) or contrib/test modules, directly on Drupal.org. See the sandbox collaboration guide for more information on that.

What if I pull/push from/to the wrong one?

It could happen before/during/after the migration that you pull or push from or to the wrong repository. git can rebase with the new repository, but unfortunately, the old commits will be mingled with the new ones.

To be able to push to the new repositories, you will need to follow instructions below to cleanup your repositories.

Rewriting history

I have taken the liberty of rewriting our collective history. There were a lot of commits with inconsistent authors: some had invalid emails, some had no real names, some had multiple different email addresses for the same person. I narrowed the list down, through extensive detective work, to a complete list of 14 distinct authors.

I have rewritten all commit authors that didn't seem right to follow a consistent pattern:

The firstname/lastname is the combination you have used in certain commits. It's not the most popular way (ie. most commits had the nickname instead of first/last), but it seems the most logical one.

The email was your most often used email or the one you have registered as such at drupal.org, if you didn't have an email address specified (in case you committed your code back in the days of CVS). (There are 6 commits from someone with a box at ovh.net that I could track, and that I can only guess are Miguel's, but I'm not sure so I have let those one fly. :)

You should make sure you have your identity set straight in your git configuration, especially your email address, if you want your contributions to be properly credited in the future on Drupal.org. In short, this means:

Again, see the manual for more information, especially how this maps to your Drupal.org account.

A fresh clone is necessary!!!!

Note that, because we rewrote history, you will need to clone your repository from new remote. So yes, contrarily to what was documented here before, you will need to clone from scratch.

We understand this may be annoying if you work on a bunch of branches or have local branches you want to keep. Unfortunately, our original tests didn't cover all use cases and were misleading in thinking we could just rebase existing repositories.

Other documentation

People unfamiliar with git or specifically git on Drupal.org should read the growing git handbook on Drupal.org.

People that are familiar with git should contribute to that manual. :)

Under the hood

The history rewrite was performed through git-fast-export and git-fast-import magic, with a fairly simple perl script of my own. I have cloned the current repo, filtered it and published a new version for git.drupal.org to pull. You can see the results here:

The exact calling sequence is this:

mkdir export orig
cd export
git init --bare hostmaster.git
cd ../orig
git clone --bare git://git.aegirproject.org/hostmaster
cd hostmaster.git
# look at the authors in the original repo
git fast-export --all |  grep -a '^author [^<]* <[^>]*> [0-9][0-9]* [+-][0-9][0-9][0-9][0-9]$' | sed 's/[0-9][0-9]* [+-][0-9][0-9][0-9][0-9]$//' | sort | uniq -c
git fast-export --all | perl /home/anarcat/bin/rewrite_authors.pl | ( cd ../../export/hostmaster.git ; git fast-import )
cd ../../export/hostmaster.git
# look at the authors in the new repo
git fast-export --all |  grep -a '^author [^<]* <[^>]*> [0-9][0-9]* [+-][0-9][0-9][0-9][0-9]$' | sed 's/[0-9][0-9]* [+-][0-9][0-9][0-9][0-9]$//' | sort | uniq -c
# rinse, repeat until the mapping is right
# push repositories online
git remote add origin ssh://gitosis@git.aegirproject.org/export/hostmaster.git
git push --all
git push --tags
# repeat with provision

This is the script source, without all the mappings:

#! /usr/bin/perl -w

%authors = ( 
    'anarcat ' => 'Antoine Beaupré ',
   # ... 
);

sub remap {
    my $a = shift;
    if ($authors{$a}) {
        return $authors{$a};
    } else {
        return $a;
    }
}

while (<>) {
    s#^author ([^<]+ <[^>]+>) ([0-9]+ [+-][0-9][0-9][0-9][0-9])$#'author ' . remap($1) . ' ' . $2#e;
    print;
}

Migration checklist

1. serve exported repositories to migration team done!
2. update the repositories with change authors done!
3. the great git migration done! git.aegirproject.org is now readonly, commits are pushed to git.drupal.org and releases are performed on drupal.org
4. update documentation in handbook in progress!
   • repositories locations done
   • crash course done
   • basic workflow done
   • release process needs review - updated for release nodes stuff, will check workflow during the next release
   • advanced workflow: branch naming conventions in progress - see 0.4-rc2 release coordination
   • migration of the install docs into the handbook work in progress
5. update the home page links done - I have changed the link in the body from git.aegirproject.org to drupal.org provision/hostmaster project pages, and pointed the "Get the source" link in the bottom to the install manual instead
6. update makefiles and files in docs/* in provision to fetch from git.drupal.org done!
7. make at least one other release done - 0.4-rc2 release coordination
8. turn off git.aegirproject.org done