Long Road to the Right Solution

Warning: This post is outdated. It is here for reference purposes only.

This is the first of what will likely be a few posts about developing BCS Group's automated backup and restore system for all the websites we manage, which is currently around 30 Joomla! sites. This initial post will cover the overview of this massive project; the next posts in the series will likely cover these general topics:

• Setup Your Dev VM (Part 1) - Initial Install of Debian
• Setup Your Dev VM (Part 2) - Network, sudo, and SSH
• Setup Your Dev VM (Part 3) - Install packages
• My site backup process
• Overview of SSH and asymetric encryption (as I understand it, I'm no expert!)
• Example setups of Jenkins CI jobs
• Unit Testing, and why you should do it
• Maybe some more as I think of it...

How It Began

I started working on this project on January 1, 2013 (according to git) and at the time, the project was to create a system that would automatically trigger Akeeba Backup on all our Joomla! client sites. The bonus was to have Akeeba send the archives to a centralized server via SFTP. So, I started building a Joomla component that would utilize the database to store some info about the sites to backup. I then developed a whole "orders" system that allowed the central system (a Joomla site with this custom component installed) to send orders to each client site when it needed to run a backup. Each client site had a customized set of scripts to trigger the appropriate backup, and then get the archive back to the centralized server. The site also stores a log and response for the central server to return, which it would then insert into the database to make a log of all the sites with their backup times, archive size, etc.

By late February 2013 I had a functional system that consisted of a user-unfriendly interface in a custom Joomla component that mostly worked if you weren't afraid of working directly in the database. While it was far from perfect, especially for anyone but me, but it gave me a sense of security knowing that I could schedule backups on client sites and they were collected in a centralized environment. It also made them easy to download the backups in bulk to have a local backup copy for testing. It was a good start, but in the best of cases I could setup a site on a backup schedule in 45 minutes, but that was only if I timed everything perfectly and didn't screw anything up. Reality was more like 1.5 hours to setup backups on a new site, and it was a complicated process. And if anything changed... oh boy. If I updated the scripts and had to load to each client site... oh boy.

The Other Half...

Now, what's the good of all these backups if we don't have time to test them? That leads to the other half of the equation, a project to restore the backups that Akeeba Backup takes to our development domains for testing to make sure the backups actually work. Having the need to automate backups and restoration led to a lot of back and forth between projects getting the backups and restoration working together. By this time it was May, and we had started planning to add a VPS from Digital Ocean into the mix. It was the right decision, but adding a VPS added to my workload, of course I didn't complain to have a new toy, but then I said "Well, the way I was approaching this problem is no longer the right way." Having the power of our new VPS made me rethink everything...

My First Mistakes

At this point I should have scrapped everything I had done up to this point, because so much had change and I really didn't have that much established... it wouldn't have been too bad. But instead what I did was try to modify my code slowly to both keep existing functionality, and completely change how it functioned at the core. Um, that's not realistic... The other mistake I made was I wasn't using unit tests. On top of these mistakes, another large project took me away from this project for close to two months, and that long of a break has a major impact on progress.

It Hurts To Grow

In October I was able to really revisit this project and made the decision to remedy the problems mentioned above. First, I started using PHPUnit and Jenkins almost simultaneously which just perfected my workflow and boosted my confidence. Using my scripts with Jenkins required some refactoring because of file path issues, which really just highlighted bugs in my code. They were bugs, they're now squashed. (Oh, I started using Mantis in March as well, another learning curve!) My code is better because of it, but again, it was more unexpected work required in order to use the tools I wanted to use.

One of the biggest problems my combined projects presented was that each project required a number of different configuration settings, and in one system I was storing config data in the database, another system used a Phing properties file for configuration and customized scripts during the build, and another used a PHP class file to store the configuration. Furthermore, each of my 3 original projects used very similar terminology in very different ways, probably because I was developing them simultaneously but they were different projects at the time, so the overlap was convenient instead of confusing.

Pain Makes Sweet Success

There are times when I wanted to trash it all, give up, and say that I couldn't do it. There were times I was so frustrated that the SCP connection worked in situation A but not in situation B even though they were virtually identical. But then when you start the build on production for the first time, hoping you don't hose it, and hoping you downloaded the backup of the production server FIRST... and then it works! That is a cool moment of triumph, and the struggle to get there makes it that much better.

To Be Continued...

My next post will cover the details needed to get setup with a Debian virtual machine using VirtualBox, and installing the various programs we will need to setup this server.