OpenX (OpenAds) 2.4.x Performance

Well, if a server is going to turn on you suddenly like a good dog gone bad I guess having something of a clue as to why is better than none. Today we had our largest spike in traffic in several months and everything held up fine, except the OpenX server wigged out shooting load up to 190! Let's see what you can do if this happens to you.

I should have known. Isn't it one of Murphy's laws, you take one day off and that's the day a server decides to wig out? Well that's what ended up happening. From my other post you'll know we recently brought our ad server up-to-date with 2.4.2 of OpenAds/OpenX.

At that time there was a bump up in the CPU usage on the webserver with the MySQL server seeming to take things in stride. Things seemed to be relatively normal since then with just that increase in CPU usages, but the patterns were close to what they were over the course of a day for the past year.

Then on this past Friday we had a huge jump in traffic compared to the norm. Well that bump caused an exponential increase in the load on the box. Started getting notifications that things weren't quite right and spent the morning trying to get things fixed.

First off we've had spikes of this nature in the past while running 0.3.x of Openads without this kind of load increase so this was pretty surprising. After trying Xcache using the source rpms from Jason Litka and enabling it not much changed. Although the amount of RAM being used by the cache went up compared to eAccelerator, which is good and as a side note you can give more memory to eAccel as well, the overall CPU usage stayed way too high.

Time to look more closely at OpenAds' configuration for this box. Everything looked right, caching is turned on and working, file store is set to be in a local directory, check. Wait, there's nothing actually *in* that directory. A frantic couple of questions to #openads on freenode confirmed that ads were being pulled from the database. Not sure if that was the problem, it couldn't hurt to move the images from the DB into the directory. Fortunately Openads under settings has a link to a script under Settings -> Maintenance -> Storage -> Move images stored inside the database to a directory.

Just one small problem it seems that over time there must be some banner records that are 'dirty', meaning although they are in the DB something isn't quite right causing the script to throw an error in lib-storage.inc.php on line 224. It seems that it is trying to delete an image from the DB if the storage type is SQL after moving that image to the filesystem. Problem is, if there isn't a file to delete the script errors out.

Here's a quick ugly fix :

if($doImages) {
$doImages->delete();
} else { echo "ERROR :".$name."";}

With that in place I've noticed only one banner that actually seemed to lose its artwork.

After dumping out the images the load on the server started to drop and settled under 0.75 for the rest of the day and into the next even as traffic remained relatively high.

Next is to upgrade to 2.4.4 and address some other admin interface slowdowns.

UPDATE Bug was reported and tagged as invalid https://developer.openx.org/jira/browse/OX-1832. Not sure how my records got screwy but I think an error message out to the user would be helpful. Anyone out there ever run into this problem and have some info on what caused it for you?

Search Google for openads openx optimization performance

March 10th, 2008