New Blog Platform: RustProof Content

The RustProof Labs blog is now being served by our own, custom built system called RustProof Content. What is RustProof Content? Well, for starters... you're looking at it now!

Meet RustProof Content!

RustProof Content is a dynamic site built on the Flask micro-framework and Python 3.3+. Content is served from almost-static HTML files that include Jinja2 tags to fit into the layouts. Being mostly HTML, it is blazingly fast especially when compared to WordPress. The HTML files (including Jinja tags) are generated from content written and stored in Markdown format, while the source code and Markdown content are tracked and distributed using Git.

The main way of deploying RustProof Content is done by using Docker images. The images are built and tested on a local development machine and then deployed to the as needed to the production server. The image is used to start two Docker containers as daemons, providing two python processes running in the background.
Nginx is then used to proxy web requests to the Docker containers, allowing me to bring a single container off-line at a time for upgrades while the other container happily serves requests. When the first container is upgraded and back on-line, the process is repeated for the second container. The end result is a completely seamless upgrade process with zero downtime. It also allows for an easy and quick way to scale out to new servers if traffic happens to increase at an unexpected rate.

With the basics of the new platform covered, let's look at why I decided to make the change.

Ditching WordPress

In the early part of 2015, we made the decision to stop using WordPress as the blog platform for RustProof Labs. This decision didn't come lightly, especially because WordPress truly is a great choice for running a blog. It's quick and easy to setup, relatively secure, and runs on almost any web hosting you might choose these days. When they say it's a 5-minute install... that's what they mean. It's easy. Building a new system from the ground up will never be as easy as that!

Reasons for Leaving WordPress

I wasn't all that unhappy with WordPress, but there were a number of factors that when combined convinced me to make the change.
The aspects I considered most important in my decision were:

• Authoring Experience
• Performance
• Security

Authoring Experience

The on-line authoring experience is the topic that nagged me the longest, but wasn't important enough on it's own to justify a change. The truth is, I hate WYSIWYG editors.
I have never used one that didn't mutilate the HTML, even after manually fixing it in the raw view. They seem to enjoy breaking things. Trying to format code blocks, pre-formatted text and blockquotes are all a nightmare. As a result, when writing in WordPress I used the "text" mode of the editor about 95% of the time. I only switched to the WYSIWYG view for a quick preview, and right back to the text view, minimizing the chance that it would decide to add or remove HTML tags.

The truth is, I hate WYSIWYG editors.

In RustProof Content, posts are written in Markdown format using whatever text editor I want. Markdown is easy to learn, and much better to work with than HTML. I don't have to worry about loosing my network connection, or accidentally navigating away (or closing) the tab I was editing in and I don't have to worry about some software deciding to reformat things on a whim. Further, the entire site is tracked and shared in a single Git repository making it a breeze to ensure that both the site and the content are the correct version.

Performance and Security

Performance and security are lumped together due to recent attack attempts against the server that powered the old www.rustprooflabs.com. To explain the attacks, I need to explain the server a bit first, but the moral of the story is that poor performance has potential to reduce overall security and resilience, while completely destroying the user experience on the site.

The blog was ran by a $5/month VPS from Digital Ocean. It had a single core processor, 512 MB RAM, and a 20 GB SSD and was powering Apache2, MySQL 5.5 and PHP 5.5 in order to serve the WordPress blog.

Note: In 2019, a $5/month Digital Ocean droplet provides 1GB of RAM and 25 GB SSD.

One of the problems with Apache2, is that it's a bit of a resource hog, especially RAM, which I obviously don't have all that much of. I had spent hours tweaking and modifying the apache config files to try to squeeze every ounce of performance out of it while keeping the server running smoothly. But... it wasn't enough.

Performance: Before

With WordPress on the described VPS in the best conditions, median response time to load the typical blog post was just over 1.3 seconds. While that isn't bad, it isn't really great either. The real problem starts when a random bot happens to find your site and floods your server with a few thousand requests to a bad URL, returning thousands of 404 pages. This can suck the life out of Apache which is forced to spawn processes for all these bad connections, consuming both CPU time and RAM, neither of which had much to spare.
It just so happens that earlier in 2015, this exact scenario was able to starve the server the powered WordPress, taking MySQL off-line and making the site unavailable for roughly 30 minutes until I was able to log in and restart MySQL manually.

After that happened, I took a snapshot of the droplet and deployed it to a new instance for some testing. I simulated the attacks using Apache's JMeter, sending 4,000 malicious requests and 100 normal requests at the test instance. It took the VPS 23 minutes to make it through those requests, each one consuming significant resources. The worst part was the median response time for the meager volume of valid traffic was up to 10.9 seconds. When I threw a few more threads at a higher rate, it was pretty easy to get the site to to slow to a crawl with response times then being measured by the minute, not by the second. Fortunately, a quick patch to the .htaccess file fixed the problem without killing performance. While that's all good and well that there's an easy fix, the real problem is that it requires a manual fix at all!

Performance: After

The VPS powering RustProof Content is nearly identical, but instead of Apache2 it's using Nginx. And no more WordPress, MySQL or PHP.

I ran the same attack simulation against the new setup, a total of 4,100 requests, and the server handled the requests in a total time of 19 seconds requiring no manual fix. During the attack attempt the regular traffic saw response times well under a second. I then did further testing at a much higher load making JMeter throw over 45,000 requests at RustProof Content. It handled the load in 25 minutes (30 requests / second) with a median response time of 179 milliseconds (0.179 seconds)! There is no way that WordPress would be able to handle that load without an upgrade to the hardware.

Final Thoughts

WordPress served me well for quite a while, but this change was well worth it to me. With close to 200 hours of time already invested in building RustProof Content, I wouldn't recommend this same path to the general blogger unless you have too much time on your hands. For me, seeing a 75% decrease in median response time is huge, and will improve the resiliency of our hardware. We're setup to handle more and more traffic, and the user experience has improved greatly. The improvement in the authoring experience is serving me well so far, but I haven't written enough this way yet to know for sure that other headaches won't present themselves.

One of the coolest side effects that came from this, is that I no longer need to invest any time or effort into backups of my blog. It feels wrong to say that I don't need backups, but it's now true! There are multiple full copies of the entire site (including content) tracked using Git; it's at a minimum of 4 distinct locations (laptop, production server, development server and BitBucket) with the full history of the project preserved.

I have written about taking backups more than once but nothing like that will be needed, at least for this blog. Goodbye, backups!

I have learned a number of new things from this project that have provided me with more topics and ideas to write about and share. Stay tuned!