Intro to the Site Maintenance project

Warning: This post is outdated. It is here for reference purposes only.

I've written before about an automated website backup and restoration tool, but I've left the overall picture rather vague so far. This post will start to remedy that problem. I've decided to call the project "Site Maintenance" but I'm not very good at naming things, so I apologize for its blandness. This post will go over the main features of the project from a "why would I want it?" point of view. Let's get started...

Why Would I Want It???

Backups are a good thing: Do you have a website that you manually back up? Do you have multiple websites that you need to backup? Do you have websites on multiple servers that you need to backup? Do you wish you could run backups daily, if only you had the time? (Or had it automated?!)

Restoring Archives is a good thing: Do you wish it were easier to test backups? Do you wish it were easier to build out multiple variations of the same main website? Do you ever wish you had an easy way to test changes in a safe environment?

Transferring Archives is a good thing: Do you wish it were easy, secure, and fast to transfer a large backup back to a production site? Do you wish you could quickly restore a backup from a centralized location?

Features and Overview

1. Regular, reliable backups
2. Restoring backups (to test they are good backups)
3. Template Sites
4. Transfer Archives

Getting regular, reliable backups is the main goal for this system. Knowing that the only good backups are tested backups, the feature to restore backups makes perfect sense to include to allow an easy way to test backups can be trusted. The other two features, templates and transfers, can help reduce some of the biggest headaches of administering multiple websites or developing new websites.

Can I Back MY Site Up?

If your site has files and/or a database, there's a good chance the answer is yes. Currently, the only tested database is MySQL (5.0+) but I will start testing backing up PostgreSQL and SQL Server databases as well. I haven't had an explicit need for anything but MySQL yet, but if you have a need that isn't currently being met please let me know in the comments.

At the time of writing (Feb. 2014) I have this system backing up 30 Joomla 2.5 websites, 1 WordPress site (this blog!), and 2 Mantis Bug Tracker instances.

Backup vs. Template

The system manages two types of archives: backups and templates. Backups are described above and are complete snapshots of a website's files and database structure and contents. Templates are semi-permanent, named archives created from existing backups. This feature can be used to provide permanent snapshots of a website, such as "MySite-InitialLaunch2014". It can also allow an easy way to restore a demo site to a known state.

What can be Restored?

The system can restore sites to predefined "targets" on the centralized VPS and currently, only Joomla 2.5 websites can be restored. I have most of the code in place to restore the Mantis BT sites, but it is easy enough for me to do that manually with how rarely that happens that I haven't bothered to allow full restoration automatically. Theoretically, any site with files and/or a database should be able to be managed with minimal enhancements required. As I said earlier with databases... if it isn't supported and you wish it were, let me know!

What's the Deal about Transfers?

I spent a whole lot of time learning how to setup secure and reliable communication between scattered servers. The method I settled on is to use public/private key pairs to allow the centralized VPS access to the production servers. This allows rsync to transfer files and I can also connect to the database (MySQL in this case) via CLI commands to backup the database. Again, what about transfers? Well, since I've already gone through so much work to allow the servers to talk to each other, I might as well make it beneficial both ways.

Currently, archives can only be sent back to the site they came from. So the system doesn't allow just any file to be sent to any server, only backups that were taken from a site can be sent back using the web interface.

Sounds Good, How Do I Start?

At this point in time the initial setup isn't point and click setup, but that is one of the issues I intend to remedy. My recommended setup does typically require having access to a dedicated server or VPS; shared hosting will not provide the features or security required for this application. I have been happily using Digital Ocean, and even their $5/mo VPS option provides sufficient abilities for a small operation. Obviously you need to have realistic expectations, and Jenkins (Java) does suck a bunch of RAM when you only have 512 MB, but it is do-able.

I will try to continuously update the official project Wiki at BitBucket, and that will typically provide the most thorough and up-to-date instructions. I recommend starting out by testing out with a virtual machine. I typically use 64-bit Debian servers such as I describe in this series: Post 1, Post 2, Post 3. For development I use VirtualBox VMs, production servers are VPSs from Digital Ocean.

My next post will cover how to start from where post 3 above leaves off, creating the file structure, setting up the keys, and everything else I end up talking about...