I wanted to start off this post by letting you know that this is NOT just another generic “How To Speed Up WordPress” article.
I’m not going to regurgitate anything that can already be found on the web. I’m not going to tell you that you should install a caching plugin, enable compression, minify your css/js etc….
After all, you should know how to do these things already. And if you don’t, you can find this generic information on hundreds of other blogs.
Instead, this article contains everything custom/semi-custom that I’ve implemented in the past month or so to speed up my own WordPress blog and basically prevent rogue accesses from bringing down my server.
And the reason it’s “little known” is because the techniques I’m about to describe will be very specific to your own blog depending on the traffic patterns you are seeing.
Note: If you have a slow blog and you really don’t want to deal with any of the technical aspects of speeding up a website, then you should probably sign up for a service like WP Engine.
These guys specialize in WordPress hosting and will make sure your blog runs as fast as possible. But naturally, this comes at a price. You should check them out if this post goes over your head:)
Anyways, before I can explain to you how and why I’ve done what I’ve done to my blog, you have to understand a few fundamental concepts about WordPress and caching which I’ll describe below.
Some Interesting WordPress Facts
Let’s say you’ve followed all of the guidelines on how to speed up WordPress already. Your blog feels zippy. Webpagetest.org tells you your blog is fast as hell. Everything is all good right? Not necessarily.
I used to feel the exact same way about my blog. After all, I follow most of the standard speedup protocols. I run very few plugins and my blog feels pretty fast under normal usage from the perspective of a human reader (The ad networks are what slow my blog down so I load ads last).
But then I started analyzing my CPU usage graphs and often noticed periods of high server load despite low to moderate traffic levels. Occasionally, my server would even become unusable or unresponsive for extended periods.
Note, the only reason why I started paying attention to these stats was because a while ago I was running my online store on the same server as my blog. And periodically I would have customers complain that the store was extremely slow. When I finally did some digging, I found that my optimized, fully cached WordPress blog was bringing my server to its knees!
Moral of the story? Just because a speed test tells you your blog is fast doesn’t necessarily mean everything is all good.
Here are some fun facts about WordPress and caching plugins like WP Super Cache and W3 Total Cache that you should be aware of.
- WordPress 404 responses are slow. Anytime your blog receives an access to a page that does not exist, your poor server has to load up WordPress, run a bunch of php code, do a bunch of MySQL queries and then spit out your custom WordPress 404 page. This is a very resource intensive task and caching isn’t going to help.
- Your caching plugin does not work very well when there are GET parameters in the URL. For example, I used to notice that my blog would slow to a crawl whenever I sent a blast out to my email list. In theory with static file caching, my server should be near invincible.
But because Aweber inserts tracking parameters in the URL, none of my files are super cached. As a result, WordPress has to generate a brand new cache file (even though it already exists), zip it up and send it out each time. The worst part is that these cached files are only used once which makes it a waste of server resources
- Rogue accesses are slow. Because rogue accesses require WordPress to load up by default, a bad bot or crawler that decides to spam your site with bad requests can take down your blog pretty easily.
- Your caching plugin might have a bug or an incompatibility with the way you’ve set up your blog. For example, for 3 years I thought WP Super Cache was doing the right thing until I started watching my logs and noticed a bug in the code. Because of the way I set things up, I had a problem where WP Super Cache was flushing my cache way too often.
The key point I’m trying to make with the bullet points above is that whenever a non-standard access happens on your blog, it uses up a lot of server resources no matter how you have your WordPress blog setup. And it’s these types of accesses that will bring your server to its knees, not regular traffic.
Detecting Rogue Accesses
The first key to optimizing your WordPress blog is understanding the traffic patterns that your blog receives. I’m willing to bet that 99% of WordPress users do not do this. Instead, they blindly follow and install various plugins and assume everything is working properly. Don’t feel bad, I was the same way.
So the first step is finding out what the heck is going on. There are many ways to do this, but the easiest way is to use the debug mode that the plugin WP SuperCache provides. What does this mode do? Basically, every time an access is directly handled by WordPress (resource intensive), it will show up in the WP Super Cache log. Here’s how you enable this mode.
Under the debug tab of your WP Super Cache Plugin, simply click on the “Debugging Enabled” check box and you are good to go!
Once logging is enabled, you can click on the “logfile” link which will point you to a file detailing your WordPress traffic. It will look similar to the text below.
15:03:46 /?utm_source=fwisp.com supercache dir:
15:03:46 /?utm_source=fwisp.com No wp-cache file exists. Must generate a new one.
15:03:46 /?utm_source=fwisp.com In WP Cache Phase 2
15:03:46 /?utm_source=fwisp.com Setting up WordPress actions
15:03:46 /?utm_source=fwisp.com Supercache caching disabled. Only using wp-cache. Non empty GET request.
15:03:46 /?utm_source=fwisp.com USER AGENT (Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)) rejected. Not Caching
The important thing to note is that any time you see something in this log, it is a bad thing because it means that WordPress has to do work. And because WordPress is a resource hog, you want it to do as little work as possible.
What To Look For In The Logs
The WP Super Cache logs are great because they tell you everything that is going on. But the sheer amount of information can be overwhelming unless you know what to look for. Here’s what you should be paying attention to in these logs.
- 404 Errors – Basically these are accesses to pages that don’t exist on your blog. Each 404 access handled by WordPress takes up a lot of server resources so you want to nip these in the bud if possible
- One Time Accesses – These are requests that cause your server to cache and compress pages that are only going to be used once (more on this later)
- Strange Bursts Of Traffic – These are usually bots that hammer your blog all at once
- Weird Caching Behavior – Are your caches getting flushed when they should? Is everything getting cached properly under all circumstances?
After keeping a log of my blog traffic for a period of 2 weeks, I discovered a lot of inefficiencies which I’ll describe below.
Bots Were Hammering My Archives
The first thing I did was look at my server logs for periods of high CPU load. Then, I analyzed my super cache logs during these exact same periods to see if there was any funny business going on. Turns out that once every other day or so, a group of bots would come and hammer all of my blog’s archive pages at the same time!
Because these archive pages aren’t cached by default, a bunch of simultaneous accesses was enough to spike my server load and makes things extremely slow. And occasionally, it even caused my server to crash.
So I took a closer look at my HTML output and noticed that I had a bunch of archive links as part of my WordPress header. Therefore, bots and other web crawlers would come by and try to browse the archives.
After a big of Googling, I found that a few other webmasters were having similar issues.
When we see loading issues with your site many times there are a large number of hits for proxy server IPs and the theory is that these proxies are cacheing your site and hitting all of those archive links. Many of the IPs we see when your site is causing loading issues are corporate, educational and Gov’t/Military proxy IPs which appear to prefetch content when someone accesses a site.
Solution: Check to see if you have the php statement “wp_get_archives” in the header code for your blog and remove it. After deleting this little snippet, all of my archive accesses disappeared.
Bots Were Accessing Non-Existent Files
The second major thing I noticed was that there were a bunch of bots that were accessing files on my server that did not exist. The problem is that each time a file access occurs, your server has to load up WordPress, execute a bunch of PHP code and then issue a 404 page.
The solution to this problem is to edit the .htaccess(Google this if you don’t know what it is) file and add the following lines.
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_URI} !(robots\.txt|sitemap\.xml(\.gz)?)
RewriteCond %{REQUEST_FILENAME} \.(css|js|html|htm|rtf|rtx|svg
|svgz|txt|xsd|xsl|xml|asf|asx|wax|wmv|wmx|avi|bmp|class|divx|doc
|docx|exe|gif|gz|gzip|ico|jpg|jpeg|jpe|mdb|mid|midi|mov|qt|mp3|
m4a|mp4|m4v|mpeg|mpg|mpe|mpp|odb|odc|odf|odg|odp|ods|odt|ogg|pdf
|png|pot|pps|ppt|pptx|ra|ram|rar|swf|tar|tif|tiff|wav|wma|wri|
xla|xls|xlsx|xlt|xlw|zip)$ [NC]
RewriteRule .* – [L]
Once these lines are inserted in your .htaccess file, your webserver will first check to see if a file exists. If not, it will issue a 404 response WITHOUT loading up WordPress.
Bots Were Accessing URLs That Didn’t Exist
What’s unfortunate about the way WordPress is written is that if there’s an access to an article that does not exist in your database, your server will have to load up WordPress, execute a bunch of PHP code and do a bunch of MySQL lookups before issuing a 404 response.
If you look at your logs carefully, you will probably notice certain patterns of accesses that you can exclude before they ever reach WordPress.
For example, I noticed that a bunch of bots were accessing my site with the URL www.mywifequitherjob.com/some-article/www.facebook.com/like.php/…. And each time, my server would load up WordPress and issue a 404 response.
So instead of having WordPress handle these requests, I added the following lines to my .htaccess file
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_URI} www\.facebook\.com/plugins [NC]
RewriteRule .* 404.html [L,R=404]
In the lines above, I’m having my webserver look for “www.facebook.com/plugins” in the URL and immediately issuing a 404 response without loading up WordPress. As you peruse your logs, you’ll find many rogue accesses like the one I described above. Write a .htaccess rule for each and these accesses will no longer load down your server.
Bots Were Hammering My Comment Links
Remember when I told you that URLs with GET parameters are handled differently by your caching plugin? I discovered that there were a ton of bots hammering away at my articles with the “reply to comment” parameter set.
When this happens (depending on your caching settings), wordpress gets loaded up and a cached/zip file is served even though it will probably never be accessed again. This is a waste of resources.
Example Taken From My Log
12:01:11 /how-to-get-more-facebook-fans-with-a-facebook-reveal-tab-or-fan-gate/?replytocom=5972 No wp-cache file exists. Must generate a new one.
12:01:11 /how-to-get-more-facebook-fans-with-a-facebook-reveal-tab-or-fan-gate/?replytocom=5972 Gzipping buffer.
The solution is to redirect all bots and crawlers with these GET parameters to the main article page. Here’s the code I added to my .htaccess file.
RewriteCond %{QUERY_STRING} replytocom
RewriteCond %{HTTP_USER_AGENT} ^(.*)(bot|crawl|spider|slurp) [NC]
RewriteRule .* https://mywifequitherjob.com%{REQUEST_URI}? [L]
This code looks for the “replytocom” GET parameter and then removes this parameter from the final URL before presenting the access to WordPress.
Crawlers Were Accessing Posts Without A Trailing Slash
I’m not sure why this is the case, but I noticed a bunch of webcrawlers that seemed to be legitimately trying to access the articles on my blog without a trailing slash in the URL.
As you may or may not be aware, a URL that is written like http://yourblog.com/article/ is different than a URL written like http://yourblog.com/article.
As a result when a URL without a trailing slash is encountered, WordPress has to step in, run a bunch of PHP code and then issue a 301 redirect to the article with the trailing slash. In order to take WordPress out the picture, you can simply insert the following rule in your .htaccess file.
# Add trailing slash
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_URI} !(.*)/$
RewriteCond %{QUERY_STRING} !.*=.*
RewriteRule ^(.*)$ $1/ [L,R=301]
By adding a trailing slash, you are issuing the 301 redirect to the correct URL before it makes it to WordPress which will save CPU resources.
I Found A Bug In WP Super Cache
Unlike most people, I don’t have my WordPress blog installed in the root of my public_html directory. In addition, the front page of my blog is not my “posts” page either. When you have your blog configured the same way I do, there’s a bug in WP Super Cache where all of your cache files could get deleted prematurely even if you preload your cache.
I’m not sure if the author of the plugin is aware of this problem, but basically I found that whenever a spam comment was issued to my blog, my entire cache would get erroneously flushed. Since I get comment and trackback spam all of the time, my cache was constantly being emptied which made caching much less efficient.
So I spent a weekend debugging this issue and finally developed a workaround. If any of you are having the same problems, let me know and I’ll show you my fix. The moral of the story here is to never assume that your caching plugin just works. You need to look at the logs!
Disable CPU Intensive Plugins
Even if you’ve followed all of my steps above, it is impossible to filter out all of the rogue accesses before they reach your WordPress blog.
Therefore, you are always going to receive hits to your site that will not be handled efficiently. There’s no way around it.
But the important thing to realize is that you most likely will not have a bandwidth problem, you will have a CPU problem.
As a result (and this may be counterintuitive), you actually don’t want to be doing anything CPU intensive like “minifying” or “compressing” of your pages on the fly. Minifying and compression helps with bandwidth at the expense of CPU usage.
The last thing you want to be doing is minifying and caching rogue accesses. In fact, you should probably consider not minifying or compressing URLs with GET parameters either.
Most importantly, you should not be running any CPU intensive plugins on your site. WP-Engine has a great list of CPU intensive plugins that you should probably try to avoid.
As I was perusing my plugin list, I noticed that I was using a “similar posts” plugin. And when I took a look at the source code, I was appalled to discover that the plugin was doing full text compares to find similar articles which is a major CPU drain and not scalable.
As soon as I find a suitable replacement, this plugin is definitely going in the trash.
Moral Of The Story
Just because an online speed test tells you your blog is fast doesn’t really mean much in the grand scheme of things.
Don’t get me wrong. Page load speed is important for the search engines and for your regular customers but you also have to account for rogue accesses that may bypass your cache and bring your server down to its knees.
So don’t just install your caching and other speedup plugins blindly. Take some time to analyze your traffic and write as many .htaccess rules as possible to minimize the work that WordPress has to perform. Good luck!
