Using a proxy cache to accelerate platform builds

While building (and re-building) platforms using makefiles and the Aegir front-end (node/add/platform) is great, it has a number of minor annoyances. They're slow, and consume lots of bandwidth, but worse still, even a small hiccup in drupal.org (or other) infrastructure can break a platform build, which requires starting from scratch each time.

When you maintain platforms with upwards of 300 modules, themes and libraries, such as with kPlatforms, and you start to deploy these to multiple clients' Aegir servers, those minor annoyances can become significant challenges.

There is good documentation on setting this up for use with Drush Make already, such as this blog post. However, there are some gotchas when doing so in Aegir. Most of the suggested solutions involve something like setting environment variables in ~/.bashrc, which works fine if you're running 'drush make' yourself in a login shell. However, since the Aegir is running in a non-interactive shell, there isn't a opportunity to set environment variables.

Thankfully, Drush Make supports curl, and opts for it automatically if it's installed. Careful reading of curl's manpage indicates that:

[curl] always [...] checks for a default config file and uses it if found.

So, in the end, all that's required is a /var/aegir/.curlrc file containing:

proxy = "http://<proxy_host>[:port]"

Thus a basic configuration on Debian would involve installing a proxy, such as Squid:

# apt-get install squid3

Then add an acl in /etc/squid3/squid.conf allowing access from your local network, e.g.:

acl localnet src 192.168.0.0/16
http_access allow localnet

Restart Squid and you're good to go:

# /etc/init.d/squid3 restart

Testing that Aegir is actually using your new proxy cache, and that Squid is actually caching the makefile projects involves taking a quick look at the logs of the Squid server:

# tail -f /var/log/squid3