NPD: What is Xoops?

Morris: Xoops is an Object Oriented content management framework written in PHP. It uses a MySQL backend and uses the SMARTY templating system. Our primary target audience is webmasters/web designers who wish to streamline website development through the use of Object Oriented, reusable code. This often comes in the form of prebuilt modules that can be installed in under 5 minutes.

NPD: So, how did you get involved with Xoops.org?

Morris: In 2003, I was looking to CMS systems to help me simplify site deployments for clients. Originally, I started with PHP-NUKE, but after many frustrating months, I found Xoops. I lurked in the shadows of the Xoops community until April of 2004. At that point, I had learned the system enough to give back to the community. Within 2 years, I was asked to serve the community as Lead Moderator and start the Moderator Team. This was also around the time that the Xoops.org servers were hacked by an ex-admin of the site and I was asked to help secure the servers and websites. This is when I joined the Server Admin Team. I've been serving in this capacity for roughly 2 years.

NPD: Why did you need to load-test the system?

Morris: Over the past several years, the Xoops.org website has been upgraded numerous times (since the 1.x branch), has had numerous customizations done to it, and it has been hacked by malicious attackers a few times as well. Essentially, the Xoops.org website is in need of being replaced.

The current website generates such an enormous load that often it will crash MySQL. Any given moment of the day, up to 400 [simultaneous] users are searching Xoops.org for information. These random search queries generate an enormous load.

Xoops.org is hosted on 1 server, which runs a full LAMP configuration. It is a dual P4 2.8GHz system with 6GB of RAM and runs on SCSI RAID1. The benchmarks indicate such a server should easily be capable of handling over a million unique visitors per month, however, we are finding that MySQL is using up the server resources. We decided to move to an enterprise class architecture, where MySQL is hosted on its own dedicated server.

NPD: So how did you (or the Xoops team) come up with the idea of load-testing the site using the Digg community?

Morris: Well, honestly, it wasn't intentional. The reasoning behind the Digg post was to get the news out to the Xoopscommunity who do not frequent the Xoops.org site. We've been working on obtaining new servers for several months now and for those of us in the Xoops community, this was a very big deal and quite newsworthy to us. While we did want as many people to visit the site and contribute to its load as possible, (hence my comments at Digg,) we were not targetting the Digg community specifically. Digg was simply a vehicle to get the news out as much of our community is tech savvy and frequent Digg.

In hindsight, posting such news on Digg probably wasn't the best idea, but it did help us to test the servers under full load. However, most of the abusive testing was done by Xoops community members using custom scripts made by our community. Simple HTTP requests are not what overload our servers. It's the MySQL queries - and lots of them.

NPD: So, what data did you get from the testing?

Morris: Essentially, we got a side-by-side comparison of the performance of the old model (single server) versus the new two server model. Before we got the community involved in the real-world tests, the servers underwent full stress testing using tools like WAPT, httperf, http_load, SQL-Bench, and crashme. It was only after all those tests that we realized that only a real-world test would simulate the load we are currently seeing.

We used the command line tool, top,to obtain information during the real world testing. As the testing was taking place, I monitored the old server and compared it with the new servers.

On average, the new Apache/PHP-only server ran at loads of under 0.5, while the MySQL-only server averaged loads of 3 and under during moderate load. Considering the abuse they were receiving, I'm very pleased with these numbers. The page generation times and overall performance far surpassed the current servers.

During the last hours of the testing, a select group of members were given the URL to a testing script and for nearly an hour, we tried to crash MySQL under loads that would make the current server fail. We were unable to crash MySQL and were still getting page load times that were twice as fast as the current servers. 

NPD:So, the new server seems to work well. What other changes will you make in the future to improve it even further?

Morris: First, we are splitting Xoops.org into three websites, a Project Homepage, Community Suport, and Addons Modules, themes, and hacks, to simplify content organization and make the sites more user-friendly. At present, all of the above is bundled together at Xoops.org and has become a massive site.

Beyond the site restructuring, we are also further tuning the configuration of the server. At present, we are roughly 80% finished with the server tuning, but there some steps that will have to wait for the new sites to be loaded on them.

The biggest improvements we have made at this point are placing MySQL on a dedicated server and optimizing MySQLs configuration and making use of features like query caching. Since our greatest load is in DB queries, and often, the terms are identical, query caching offers a tremendous performance boost.

NPD: What about the application itself? Any improvements there?

Morris: Xoops.org runs a version of Xoops that you will not find available to the public. This is due to numerous upgrades over the years and multiple core hacks. With the site restructuring, we will be migrating the Xoops.org websites to the latest, stable branch of Xoops. The latest versions of Xoops are much more efficient than their predecessors and do not place as much load on a server.

Technorati Tags: performance+testing digg xoops web+performance mysql enterprise+computing