Add a Comment Now - We Want to Hear From You
*Update on Wafaa Bilal and Recreational Network Traffic
The Brain Behind Paintball Artist Bilal: ‘Domestic Tension’ Sysadmin Jason Potkanski talks about how he helped keep the paintball-via-webcam site running.
Jason Potkanski, who works at the Citizendium Foundation, read the Chicago Tribune Daywatch, to find Wafaa Bilal’s site, Domestic Tension. He looked at the site and saw that it was getting swamped over a DSL link. He sent Ben Chang, lead software programmer of the project, an e-mail offering help, and got an answer back the next day. He visited the project site at the Flatfile Galleries on the 11th – a day when the gun was down most of the day due to the strain that the Internet was placing on the connection – the DSL line and commercial-grade router they used were taking a beating from the Slashdot-like effect of a small pipe.
He immediately offered some performance optimization techniques, such as setting keepalives to 3, and installing eAcellerator to speed up the PHP script, replacing the memory inefficient tail function for the chat. And of course, he told Bilal that he would need a dedicated line to the Internet, and by the next day, they got an OC3-quality connection to Steadfast, at Equinex in Chicago. Immediately he saw an improvement.
He also mounted the webcam image and chatlog/shooter log via NFS, remoted in to do the mounts and copy the application over to the Steadfast connection.
Another problem was the use of XML requests, which wasn’t viable on all browsers, and the fact that the original configuration kept three separate requests open for the chat, the shooter log, and the web cam.
If he had the system to design all over again, he’d design it to use a backend database for the logging features, combine the shooter log and chat so that the application requires only two keepalive connections, not three.
(Continued...)
“When the site was Digged,” Potkanski said, “that was a struggle with the server which had only 1 GB of RAM. I turned keepalive off and bumped the maxclients up to 384.” That was too high, however, and ran into swap thrashing. “Remote power cycling is very useful,” he said.
“As soon as the story got on the front page of Digg, the JavaScript applications stopped working, basically. That’s a big reason not to ever, ever, ever use XMLhttprequest,” he said. “At that point, the chat and shooter logs were hit or miss – it was a lost cause with that many visitors – but as for the primary goal of keeping the gun and gun movement working, the site worked fine. The web cam was a bit sluggish and choppy, but you could still see an image every 15 seconds or so.”
At one point, the traffic was so heavy that it literally set the DSL line on fire. First, a funny smell came from the consumer-grade router, which was fixed through power cycling. It simply couldn’t handle the IP table with thousands of clients. Then the wires for the DSL/Ethernet ran up the wall by Wafaa Bilal’s desk “in the conflict zone” and the paint from the paintballs leaked down the walls into the basement and onto the DSL line.
“Two plus two equals doom,” Potkowski said, succinctly. “Maybe someone needs to do a YouTube video on whether paintball paint is conductive.”
If the problems with performance under unusual conditions weren’t enough, they also had problems with security, which became apparent when the site started to get popular.
“The implementation they originally had for the gun exposed the shoot method in the index.html file,” Potkanski said, “so by reading the source, people were able to bypass the javascript timer and click the link directly. So imagine some jerk going ‘click, click, click,’” essentially firing the gun, which is set to a reasonable “rifle-like” rate, like a machine gun. He fixed this by wrapping it in a forward.php file. It didn’t prevent people who already knew the command from machine gunning, so eventually he had to write blocking code. “I’ve been on the Internet since 1995. Kiddies are kiddies,” he said. “You give ‘em a gun hooked to a Web form, they will find a way to abuse the hell out of it.”
In addition to those who were hacking the site to launch as many shots at Wafaa as possible, there were those who were hacking the site for more “humanitarian” purposes to try to prevent harm to Wafaa.
“Very early on,” Potkanski said,” someone posted a bash script that used wget to continually get the camera to aim to the extreme far left. We blocked the wget referrer, but then some knucklehead emulated the referrer field in wget. We handle any ‘automated twit-age’ by banning them. I provided Wafaa with a quick and dirty Web interface to drop IP tables in case of an emergency.”
Traffic is likely to increase as Ben Chang and Jason Potkanski are trying – no guarantees – to bring audio on board for the final day of the project, using a combination of QuickTime and Darwin Streaming Server – if they can find a way to get the outgoing bandwidth down.
More important than the site’s traffic, an examination of the Web stats for the server showed that from May 15th to May 30th, a two week period, 67,747 shots were fired. Potowski promises that he’ll concatenate the gzipped logs from the period once he has a free moment.
Overall, Potkanski thinks the project is “equal parts insanity and brilliance – and of course, scale naivety. It reminded me of the early days of Slashdot when webcam links in random places would bring down servers around the country. Aspiring artists and people who plan on becoming popular had better plan for problems of scale.”
Update: (6/4/07) Last Sunday, Wafaa Bilal's site was attacked with a bot flood, which caused Jason Potkowski to implement a banning procedure that eventually netted over 130 banned IP addresses. Additionally, the gun's timer has been changed so that individual users have to wait longer between taking shots, and he and Ben Chang did indeed get audio working for the final day of the project.
[Ed Note: When this article was originally published, the name of Jason Potkanski was misspelled as Jason Potkowski. We regret the error.]
More Information:
--------------------------
-Recreational Network Traffic Calendar
