Importing sites

Tagged:

It is possible to import existing sites into the Aegir hosting system, so that they can be managed by Aegir.

These instructions assume you've set up a new server with Aegir on it, and you want to import sites into Aegir from another server, or even from the same server, using backups.

There are basically three ways of importing sites in Aegir:

This section also regroups troubleshooting and other less orthodox techniques.

Importing from existing Aegir backups

Tagged:

If you need to import a site from an existing Aegir system, this can be done from an Aegir site backup.

Requirements:

  1. New Aegir version 0.4 alpha15 or later.
  2. Access to Aegir backups from the old server.
  3. A platform set up and verified that matches the platform on the old Aegir server.

1. Copy backups

First log into the new server via ssh - switch to the aegir user. Create a temporary directory somewhere, for example:

mkdir /var/aegir/tmp/migrate

Run a command to syncronize the new directory with the backups directory of the old Aegir hosting system. This command does the entire directory, but you may want to modify this if you have a lot of unneeded backups on the old system.

This command also assumes that you can shell into the other server as the aegir user. Modify as required.

rsync -aPv -e ssh aegir@old-server.com:/var/aegir/backups/* /var/aegir/tmp/migrate

Now you can re-run the above command just before you are going to migrate a site. This is handy if you have a lot of sites to migrate and each site requires attention. Simply make a new backup of the site in the old system, and rerun the rsync command.

2. Create the site context and deploy the backup

For brevity, I assume that drush can be executed as drush. Replace with php /var/aegir/drush/drush.php if needed.

Pay attention to these two commands, they require modification to match the name of the site, the alias of the platform, and the name of the backup archive. These commands should each be on one line. You also need to know the platform alias. You can see all the platform aliases available by typing drush sa | grep platform.

First tell Aegir about the site, which creates a site alias.

drush provision-save @example.com --context_type=site --platform=@platform_d6 --uri=example.com --db_server=@server_localhost --client_name=admin

Then run the provision-deploy:

drush @example.com provision-deploy /var/aegir/tmp/migrate/example.com-2010-11-11.tar.gz

Note: if you receive an error, you might also try adding the site to your new Aegir platform (using the same platform and site name, of course). Then run the provision-deploy again.

3. Verify the Platform

The final step is to import the site into Aegir. This is easily done by logging into the Aegir front-end and verifying the platform, which then will trigger an import task for the new site.

4. Caveats

The provision-save command will assume the database server is the same as the master server and that you are not using any install profile. So if your DB lives in another server or if you are using an install profile like openatrium you will probably need to edit the file created at /var/aegir/.drush/example.com.alias.drushrc.php

5. Setting a Non-Default Install Profile

If you are using a non-default install profile, you may want to specify your install profile by adding --profile=profilemachinename to the drush provision-save command above.

Using the example above but using an install profile for Open Atrium, the command would look like:

drush provision-save @example.com --context_type=site --platform=@platform_d6 --profile=openatrium --uri=example.com

Importing a single site manually

Tagged:

If you already have a platform setup on Aegir with EXACTLY the same codebase as your existing site, then you don't need to transfer the entire old codebase - you can just transfer the sites/example.com directory. However, you also need to make sure any dependencies on contrib modules that your site has, are covered on the codebase or Platform that you're importing it into.

In general it may be considered safer/less prone to confusing errors to transfer the entire old codebase into Aegir as a whole Platform, whereby the site will be imported automatically under that Platform. You can then migrate it in Aegir to one of your existing Platforms later. See importing from a complete Drupal platform for this method.

Also, it is much faster to just "deploy" an Aegir backup than go through the manual procedure here, so you should generally follow that procedure instead of the one defined here unless you have a very hairy setup.

To transer a site manually into Aegir, those are the steps you should follow:

1. Create the site database, user and directory

In order to import the site, you need to create a database and database user for the site, along with a directory in the platform.

The simplest way to do this is to create an empty site in the right platform and overwrite it. Through the commandline:

drush provision-save @example.com --context_type=site --platform=@platform_d6 --uri=example.com --db_server=@server_localhost --client_name=admin
drush provision-install @example.com

Through the UI, this can be done simply by creating a site in Create content.

If this is not working for some reason, you can create the mysql user and database manually using the following:

GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, INDEX, ALTER, LOCK TABLES, CREATE TEMPORARY TABLES ON databasename.* TO 'databaseuser'@'localhost' IDENTIFIED BY 'password';
CREATE DATABASE databasename;

2. Transfer the files

You need to copy the sites/example.com directory from your old server to the new Aegir server.

To do this, first create a tarball of the site on the old server. Within the sites/example.com directory type:

tar -zcvf example.com.tar.gz .

Then, on the Aegir server, you can create a directory for it in the codebase (platform) you want to put it under:

cd /var/aegir/platforms/drupal-X.Y/sites
mkdir example.com
cd example.com
wget http://example.com/example.com.tar.gz
tar
-zxvf example.com.tar.gz ./files ./modules ./libraries ./themes

Instead of using wget you could of course use SCP or FTP to transfer the tarball onto the Aegir server.

In the above we assume that you have created an empty site as directed above, if not, you will also need to extract the settings.php file, so that last command should instead be simply:

tar -zxvf example.com.tar.gz

Then you will need to modify the settings.php file to follow the user and database created in the first step. Again, this is not necessary if you let Aegir create the site for you.

Once the file is unpacked, check the ownership and group of each file by using ls -al. It should be either 'aegir:aegir' or 'aegir:www-data'. Change it if necessary. In particular, files that may have been uploaded on your site via modules like imagecache, upload, Profile pics etc, may need to have their ownership changed:

sudo chown -R aegir:www-data $platform/sites/*/files

3. Transfer the Database

Make a backup of the database on your old server (Backup and Migrate module is good for this, or you can use phpmyadmin or your favourite MySQL management tool. It's a good idea to truncate the cache, search and watchdog tables first to reduce the size of the database. The "Backup and Migrate" module does this for you.

If you are importing a site from a previous Aegir installation somewhere, such as from a backup tarball, the database.sql will already be within the tarball you unpacked. You can use this to import into a new database on your new server.

You can import the database with drush using the following command:

drush @example.com sqlc < database.sql

4. Verify In Aegir

In the Aegir web interface click on the name of the platform you have added your site to. You can access the list of platforms via the 'Platforms' tab in the primary links menu.

Execute a new Verify task on this platform via the 'Verify' button in the task list.

Aegir will now re-verify the platform. In the process of this, it will discover your new site and spawn a new 'Import' task for the site.

Once it's done this (it may take a couple of cron runs to dispatch these tasks in the queue) the tasks should turn green (if not, see 'Troubleshooting' below).

At this point, presuming you have updated DNS or are overriding DNS via an entry in your /etc/hosts file to access the site via the new Aegir server, you can visit the site in a browser and check around to see that it has worked. Pay particular attention to any links in node content that pointed to paths referencing /sites/community.aegirproject.org if your site was not part of a multisite setup. (Drupal has a habit of storing these paths in the database, or they may have been hard-coded into nodes by users).

Aegir will not go through all your database and update all URLs, so some images or links may be broken. This will need to be done manually. Aegir does attempt to update paths stored in tables such as 'files' though.

It's a good idea to clear the caches, and you may need to get imagecache to rebuild its thumbnails if you use it.

Importing a complete Drupal platform

Tagged:

This way of importing sites is often considered the safest whereby you transfer across an entire codebase containing Drupal core, and define it as a Platform first. This makes sure you bring any dependencies, contrib modules etc, in exactly the same version and configuration that is referenced in the database for the site.

1. Transfer the codebase

Create a tarball of the codebase on the old server. Within the drupal root directory type: tar -zcvf example.com.tar.gz * .htaccess

Note that we have to specify including the .htaccess because it isn't picked up by the * wildcard due to being prefixed with a full stop. It is important to bring along the .htaccess for a platform so that Aegir can consider these settings when it generates Apache configuration for this platform.

Then, on the new server you can create a directory for it in the codebase you want to put it under:

cd /var/aegir/platforms
mkdir your-new-platform
cd your-new-platform
wget http://example.com/example.com.tar.gz
tar
-zxvf example.com.tar.gz

Instead of using wget you could of course use SCP or FTP to transfer the tarball onto the Aegir server.

Once the file is unpacked, check the ownership and group of each file by using ls -al. It should be either 'aegir:aegir' or 'aegir:www-data'. Change it if necessary. We have to assume you have basic knowledge of Linux and POSIX permissions - we can't be responsible for teaching you sysadmin skills :)

In particular, files that may have been uploaded on your site via modules like imagecache, upload, Profile pics etc, may need to have their ownership changed: sudo chown -R aegir $platform/sites/*/files

If your site previously resided in sites/defau1t you need to move it because Aegir ignores the default directory (it has no way of understanding what URL this would be accessed by, so it is impossible to 'manage' it). Each site needs its own directory with the correct domain in typical Drupal multisite design:

mv sites/defau1t sites/example.com

If you do this, you may need to also update your filepath's stored in the files table by running the following query on your database:

UPDATE files SET filepath = REPLACE(filepath, 'sites/defau1t', 'sites/example.com');

There are many other places in the database that can suffer the same problem, the node_revisions table for example. If you have phpMyAdmin available you can easily search the entire database for 'sites/defau1t'. Select the database, then click the Search tab. Once you know which tables are affected you can run a variation of the above query to correct these records.

2. Transfer the Database

Note: This step is not necessary if you are moving your site from one directory on your server (e.g., /var/www/html) to a newly created Aegir directory on the same server (e.g., /var/aegir).

Make a backup of the database on your old server (Backup and Migrate module is good for this, or you can use phpmyadmin or your favourite MySQL management tool. It's a good idea to truncate the cache, search and watchdog tables first to reduce the size of the database (Backup and Migrate does this for you).

Aegir does not provide support for importing databases with prefixed tables so it is best to remove prefixes form the database. However, there is work to bring this support in (http://drupal.org/node/483346) and a solution if you want to keep prefixes on your tables (http://drupal.org/node/483346#comment-3113516)

Unlike traditional Drupal projects the database settings cannot be found in settings.php. The database credentials are stored in the Apache vhost config of the associated site. This is a security measure implemented by the Aegir project.

In the vhost directory on your old server, /var/aegir/config/server_master/apache/vhost.d/, is a vhost file for every site on your server. The contents of the file can be displayed in your terminal with the command cat thenameofthefile. At the top of the file you will find the credentials needed to create a database on the new server.

On your new server, manually create a new database and upload the .sql file from your backup.

Then create the mysql user that your site accesses the database as, and grant it all permissions on that database except 'GRANT'.

Connect to the database on the old server: mysql -u root -p

Create the new database: create database yourdatabasename

Add the user: CREATE USER 'yourdatabaseuser'@'localhost' identified by 'youruserpassword';

Give the necessary privileges: GRANT ALL PRIVILEGES ON yourdatabasename.* TO 'yourdatabaseuser'@'localhost';

Import the database: mysql -u yourdatabaseuser -p -h localhost yourdatabasename < thenameofthedatabase.sql

3. Setup Platform in Aegir

In the Aegir web interface click on 'Create Content' and 'Platform'. This step is often easiest when using the admin-menu on the top of your screen.

Enter the name you want to use for the platform (eg 'My Platform import'), and the path to the platform on the server (eg '/var/aegir/platforms/your-new-platform')

Click Save to submit the form, and Aegir will spawn a Verify task for this platform into the queue.

Once the task is complete this the task should turn green (click through to the homepage to refesh and check. If not, see 'Troubleshooting' below).

4. Import and Verify Sites In Aegir

Upon completion of the above Platform verification task, Aegir will discover your new site and spawn a new 'Import' task for the site.

Once it's done this (it may take a couple of cron runs to dispatch these tasks in the queue) the tasks should turn green (if not, see 'Troubleshooting' below).

At this point, presuming you have updated DNS or are overriding DNS via an entry in your /etc/hosts file to access the site via the new Aegir server, you can visit site in a browser and check around to see that it has worked. Pay particular attention to any links in node content that pointed to paths referencing /sites/community.aegirproject.org if your site was not part of a multisite setup. (Drupal has a habit of storing these paths in the database, or they may have been hard-coded into nodes by users).

It's a good idea to clear the caches, and you may need to get imagecache to rebuild its thumbnails if you use it.

5. Migrate To Another Platform

Now that you have your sites under Aegir's control you can take advantage of its power, and easily migrate them to another platform. In the event that your sites were on old versions of Drupal core and a new one is available and present as another Platform on your Aegir server, you can use the Migrate process to upgrade those sites onto the new core.

For details on how to Migrate sites, consult the Migrate documentation.

6. Troubleshooting

For those who are migrating existing sites that live with their files in the sites/community.aegirproject.org folder, things can get tricky. However, you can follow this tutorial and successfully get things moved over rather easily.

Setting up a test Aegir site from an existing one

We have already seen how to import sites in an Aegir install, but what if we want to test the "hostmaster" site itself, ie. setup a development or staging environment of an Aegir install itself?

This page explains how. The basic steps are:

1. Installing aegir

The first step should be fairly simple and intuitive if you got that far: you have to install aegir itself on the staging server. You should install the specific version that is currently in production.

Maybe that also involves uninstalling the existing install.

2. Backup the production site

Run a backup on the production site, on the production server. This can be done in the frontend or in the backend using:

drush @hostmaster provision-backup

3. Deploy the production site in staging

Copy the resulting backup to the staging server. Then you should be able to deploy it with the following command:

drush @hostmaster provision-deploy --old_uri=aegir.koumbit.net /home/anarcat/aegir.koumbit.net-20110326.161420.tar.gz --debug

In this case, we use deploy to also rename the site (--old_uri=aegir.koumbit.net).

4. Tweaking the install for the staging server

The environment on the staging server could be very different from the production server. The server name will be different, maybe there will not be a dedicated mysql server, platforms will not exist, etc.

It is recommended to review the "Servers" page to change the domain names to the staging environment. In Koumbit's case, this means:

  1. editing the master server to change the hostname and IPs
  2. renaming the DB server to "localhost" and changing the database credentials
  3. renaming the main hostmaster site:
    drush @hostmaster sqlq "UPDATE node n JOIN hosting_context h ON h.nid = n.nid SET n.title='hostmaster.koumbit.net' WHERE name='hostmaster';"
    drush @hostmaster sqlq "UPDATE node_revisions nr JOIN hosting_context h ON h.nid = nr.nid JOIN node n ON n.vid = nr.vid SET nr.title='hostmaster.koumbit.net' WHERE h.name='hostmaster';"
  4. change the sites to point to the good web and db servers:
    UPDATE hosting_site SET db_server=3631 WHERE db_server <> 3631;
    UPDATE hosting_platform SET web_server = 3 WHERE web_server <> 3;

Note that this will have the added benefit of reverifying all platforms and therefore create them if they are makefile-based.

5. Troubleshooting

I had weird issues with failing to run updatedb on the hostmaster site (while deploying to another alias would work). My workaround was to suspend the process right before it would run drush updatedb, then run updatedb manually, twice:

drush @hostmaster updatedb
drush @hostmaster cc all
drush @hostmaster updatedb

Troubleshooting creating a platform or importing sites

First things to check:

Does the aegir user have proper permissions?

sudo chown aegir -R PLATFORM/sites/yoursite

Is the database properly set up? Try drush sql-cli to see if the site db user has access and if there are tables (try SHOW TABLES).

If something goes wrong with importing a site you might end up with a node for a non-working site. In such a case it can be good to delete the node for this non-working site and verify the underlying platform again. Since there is no direct way to delete such a node from the user interface you have change the URL from node/NUMBER/edit into node/NUMBER/delete to do this.