A number of East Coast based customers of World of Warcraft have been experiencing connection delays and uncomfortable lag – and no one seems to know exactly where the problem is.

The New York Post says that Blizzard is blaming Time Warner Cable is for the problem:

"The only commonality between all the players experiencing these disconnects and extreme latency is Time Warner/Road Runner," the company said in a June 23 support post.

But the Digital Communications Director for Time Warner has said that the lags and disconnections are not on their end and points to the traceroutes as evidence.

Take a look at some of the traceroutes posted to the thread in question ... starting here, at comment #446: http://tinyurl.com/5gqe27

If you follow the commenter's posted trace results, you'll notice that it's only on TWC's Roadrunner (rr) network for the first 6 hops — with maximum response times of 10 ms. The response time jumps drastically at hop # 11 — when the trace is no longer on the Roadrunner network.

Scroll down further on the same page to comment #456, and you'll see something similar — a giant leap in lag times. However, this trace never touches our network. It starts at Verizon, goes to Alter.net at hop #5, and then jumps to ATT.net's network at hop #8. Hop #9 shows a response time of 114 ms — quite a jump from the 49ms at hop #8.

So, what’s going on?

One of the theories is that Time Warner is lying and is throttling World of Warcraft traffic, considering all the bad blood between savvy broadband users and major ISPs over BitTorrent throttling. And while I can’t prove that they’re not doing so, I have to admit that the theory doesn’t seem very likely because of the nature of World of Warcraft.

See, MMORPGs care more about latency than bandwidth. While patch downloads can be huge, the majority of the content of WoW requires low latency to provide instant responses to actions. Latency, in WoW can result in an annoyingly choppy game, and a multi-hundred millisecond delay may be the difference between slain dragon or hobbit pâté.

So from a bandwidth-saving perspective, a ISP wouldn’t have a whole lot of motive in blocking World of Warcraft or other MMORPGs.

Additionally, Comcast, Time Warner, and other cable companies were rumored to use BitTorrent throttling because both legal and copyright infringed video files competed with the standard television cable offerings of those companies. This also doesn’t seem to be the case – as while more generally, time spent playing WoW is time not spent watching TV, it’s not a specific competition. Indeed, MMORPGs are one of the key drivers for broadband speeds in the U.S., and I have trouble believing that TW or any other company would knowingly interfere with such a cash cow.

Indeed, I believe that TW might be reaching out to users to find out more about the problem because TW might be interested in solving the problem instead of losing customers to other ISPs like Verizon FIOS.

Of course, I don’t know anything – and I wish that I had some inside information to figure out what was going on and solve the problem. Not only would I look like a genius but every one of my friends who plays World of Warcraft would hoist me on their shoulders, and treat me like a Lich King for a Day. Sadly, I think that it’s going to take Blizzard and TWC together to try to triangulate why this problem is happening.

I’ve been getting a number of e-mails and comments asking why I haven’t yet written anything about the Russian/Georgian war and the supposed “cyber-warfare” taking place. ZDNet has written extensively about the DDoS attacks being waged against Georgian government sites.

At first, I thought that this was solely a security issue. As a general rule, I don’t like to talk a whole lot about computer security on Network Performance Daily because I lack the proper mindset to get around security – security experts are people who look at things and see how to break them down, network performance experts are generally people who look at things and see how to build them better. Besides, there are tons of blogs out there about computer security, and very few about network performance.

I’m not going to get into the geopolitical aspects of it, except to say that getting involved in a land war in Asia is one of the “classic blunders.”

However, I did start thinking about things… I mean… wasn’t the Internet partially designed to be a resistant form of communication in case the Russians ever attacked? The irony of the Russians effectively taking down a country’s Internet is… well, it’d be funny if it wasn’t for all the people dying.

What this does tell me, however, is that cloud computing (and I’ll continue to call it that despite Dell’s claim to the term,) has a long way to go. While the Internet can be cheaper and simpler than having a fully-fledged IT department monitoring in-house servers and applications on leased lines over a WAN, the one problem that in-house IT has licked is fault.

For the most part, we’ve managed to get it so that we no longer worry about fault on the enterprise network. It was a while ago that we passed the 99.999% uptime mark. So while we may worry about security and performance, we typically don’t have to worry about the network not working.

But cloud computing still has fault problems. And it doesn’t take the Russians attacking. I love Stumbleupon, but they went down for a few minutes yesterday – Twitter also, but they’ve got problems. Even Gmail, which I greatly rely upon for my personal e-mail, went down for a little while earlier this week.

By and large, cloud computing makes great solutions for smaller companies and start-ups because of the low cost, low maintenance, and portability. However, the tradeoff is reliability – Internet applications simply aren’t as reliable as the bulky solutions that get things done when a single hour of downtime can mean thousands in lost business.

There really is no such thing as a private cloud. The entire concept revolves around using IT services offered from outside companies, which connect on public lines through to shared servers.

This is not to say that there is no room for the cloud in enterprise computing but that incidents like the South Ossetian war show that Internet applications suffer from one fatal flaw: They’re on the Internet.

One of the things that we hear most often is that it’s hard to identify the problem being the application, server, or network.  We can now officially add to this list: Application, server, network, or virtualization layer. 

Virtualization is to IT sorta what light is to physics.  Is it a wave?  A particle?  Is it a software thing?  Is it hardware?  Virtualization is… virtualization.  It’s something different, something new, and something which is very very difficult.  Also like physics.

And when things go wrong with virtualization, they can go wrong very very badly. According to Network World:

Many VMware customers Tuesday were prevented from logging onto their virtual servers as a bug distributed in a software update effectively stopped the boxes from powering up….

"This certainly appears to be the most publicized bug for VMware so far, and I think it is damaging to VMware and virtualization as a whole. The hypervisor is the lowest software level on the server and if you have an issue like this, boom, all your infrastructure is down," says Gary Chen, a senior analyst with Yankee Group. "Software will always have bugs, but a widespread issue like this that affects all VMs is really damaging, especially at this point in time where virtualization is starting to take off. VMware is going to have to fix this fast, provide an explanation, and outline what they will do to strengthen their QA in the future."

The bug is preventing users from powering on virtual machines, though VMs already running continue to do so.  There is a workaround – set the date back, turn on the VM, reset the date – but that can be tricky when certain applications need accurate timing information  – banking records, for instance.

Of course the fact that it’s complicated means that it’s even harder for lay people to understand. I’ll bet you that even though the real problem is the virtualization layer, end-users who find this problem prevents them from working blame the network.

by Brian Boyko
Editor, Network Performance Daily

USA Today publishes “CyberSpeak” from columnist and radio talk-show host (not to be confused with “talk radio show host”), Kim Komando. For over a decade now she has been helping people become more comfortable with digital technology and the Internet. She has won the 2007 Gracie Award, and is a journalist I greatly admire.

I give her that introduction, because I’m going to rip her latest USA Today column, entitled “Web Delivers New Worry for Parents: Digital Drugs,” to shreds, turn the shreds into mulch, and turn the mulch into compost.

We all know that music can alter your mood. Sad songs can make you cry. Upbeat songs may give you an energy boost. But can music create the same effects as illegal drugs?

This seems like a ridiculous question. But websites are targeting your children with so-called digital drugs. These are audio files designed to induce drug-like effects.

All your child needs is a music player and headphones.

The article goes on from seizing the “maternal fear gland” by the throat to explain that she’s talking about binaural beats, which supposedly affect your brain waves and give the listener a high not unlike taking a drug. If this sounds familiar, it’s a lot like the plot behind the William Shatner-created “Tekwar” series of novels.

(Continued...)

Continue reading "Won’t somebody think (better) of the children?" »

A study commissioned by VoIP provider Inclarity and conducted by YouGov among workers in the United Kingdom found that up to 60% of those polled had, during the previous year, had experienced a full day’s disruption of their company’s phone system. And according to SC Magazine UK, telephone downtime costs UK businesses around £14,000 each day, or, with the current exchange rate, about three billion U.S. dollars per minute.

Worse still, 61% of respondents didn’t have, or weren’t aware of, a disaster recovery or business continuity plan in the event of phone service problems. Considering that kind of money, it is extremely important to have a backup plan for the phones, be able to isolate performance issues, and speed recovery times. Because the only thing worse than one day without phone service is multiple days without phone service. Or baby cheetah murder. I suppose baby cheetah murder is worse.

VoIP is sensitive to latency and jitter, and anything that interferes with either latency or jitter will create bottom-line impacting problems. Now think about all the possible things that can increase latency or cause jitter either within your network or outside of your control. Sobering thoughts, huh?

Monitoring VoIP performance and being warned before problems impact the bottom line is very important, but sometimes, things like hurricanes, tornadoes, backhoes, and little kids pushing the bright shiny red button will happen. it always helps to have a backup plan in place of some sort of catastrophic failure.

IT World has published a fascinating look at the “Nerdlympics” – nerdy alternatives to the blah in Beijing. A neat read, it looks at the world of speedcabling, obfuscated code contests, Soldering, Buzzword Bingo, and Speedrunning Super Mario Brothers.

There are of course, a few things they left out.

For example, there’s a documentary about CalTech’s Basketball team, called “Quantum Hoops.” It details the team’s goal to actually win a single game, despite the fact that the team has more valedictorians than high school varsity basketball players.

Of course, CalTech’s Basketball team is WAY less nerdy than Princeton and Middlebury’s Quiddich league.

“The role of the snitch is filled, not by an enchanted, plum-sized, golden-feathered ball, but by a hyperactive college student dressed in yellow with a penchant for running and wrestling. To capture the snitch, one of the seekers must grab a black sock hanging from the snitch’s shorts.”

I have found it impossible not to laugh at that previous sentence.

Of course, ultimately, there’s not much difference between Geeks and Sports Geeks…

The Wall Street Journal has an article out about “Why the Olympics Scare Tech Pros.”  But really, should this even be scary anymore? 

We’ve known for quite some time that major cultural events, such as the Olympics, can increase recreational traffic on the network as people tune in to catch sporting events.   These events can generate enough traffic to push many enterprise networks to the limits and adversely affect business-critical application performance.  And NBC plans to stream footage of the Beijing games over the Internet.

There are a number of solutions including QoS policies, limiting bandwidth to certain subnets… I particularly like the approach that Brunswick (the bowling guys) are taking.

[Cathy] McClain [divisional chief information officer at Brunswick] can’t just block streaming videos. Some Brunswick employees, the marketing department for example, have to watch the Olympics for work reasons. And blocking sites doesn’t fit with the company culture. Instead, she’s letting workers do whatever they want. But if the network becomes strained, a message will pop up on employees’ computers asking whether they’re watching the video for work-related reasons, and if not, could they please wait until off-peak hours.

The messages explain that Brunswick is trying to save money and McClain includes her phone number so that anyone who has a question can call for an explanation. And they don’t block the video – they just ask workers if they have to watch right now.

It’s a backlash-free way to protect the network. “My community is polite,” McClain tells us. “They get it.”

So, yes, there needs to be policies in place for this sort of thing. But it’s not like this is any sort of big surprise.  We’ve had four years to prepare for this.  Four.  Years.  And chances are if you’re reading this you know about what streaming video can do to your network if left unchecked, you’ve probably lived through a few March Madnesses and Super Bowls and World Cup and World Series and Shriner Bowls…

Besides, the Olympics are crap.

What?  They are! 

First, and to the chagrin of those guys at Brunswick, there are no bowling events.  They just completely ignore the sport.  How can you even take the Olympics seriously if they don’t include bowling?  We’re talking about a franchise whose winter version has included curling.  Curling is practically the same thing, only colder and with brooms. 

Secondly, the International Baseball Federation (IBAF) is changing the rules of baseball at the last minute.  You can’t do that!  You can’t really even call it baseball if you change the rules.  Call it… I don’t know.  Whinyball. 

And of course there’s the whole China/human rights thing. 

Worst of all, the Olympic games in Beijing is pretty much dominated by sports.  Seriously, someone should talk to their marketing department.  I feel pretty confident based on informal polling of myself and my friends at the Linux User Group, the guys at the comic book store, and my LARP buddies – and they pretty much agree that the Olympics has to have some sort of draw other than sports, because really, who likes watching that stuff? 

The Antikythera mechanism was discovered on 17 May 1902, when archaeologist Valerios Stais noticed that a piece of rock recovered from the Antikythera wreck had a gear wheel embedded in it. Examination revealed that the "rock" was in fact a heavily encrusted and corroded mechanism that had survived the shipwreck in three main parts, and dozens of smaller fragments. It was inscribed with a text of over 3,000 characters, most of which have only recently been deciphered. These were part of a manual, which describes how to set up the instrument and how to use it for observations, with references to the Sun, the motion of the planets (stationary points), Aphrodite (Venus), Hermes (Mercury), and eclipses.

It is evidence that the ancient Greeks of 100 B.C.E. had access to mechanical clockwork technology not rediscovered until the 1700s. It is considered the “first computer.”

And now, the story of the Antikythera Network, with equal parts fact, historical accuracy, and pizza.

Dr. Tim Wu has recently penned an editorial to the New York Times entitled “OPEC 2.0.” which argues that we need an alternative source of bandwidth much like we need alternative sources of energy.

It’s the sort of editorial which causes other tech columnists and blog editors to immediately loathe themselves because it’s obvious in retrospect, but Dr. Wu thought of it first.

Just as the industrial revolution depended on oil and other energy sources, the information revolution is fueled by bandwidth. If we aren’t careful, we’re going to repeat the history of the oil industry by creating a bandwidth cartel.

Like energy, bandwidth is an essential economic input. You can’t run an engine without gas, or a cellphone without bandwidth. Both are also resources controlled by a tight group of producers, whether oil companies and Middle Eastern nations or communications companies like AT&T, Comcast and Vodafone. That’s why, as with energy, we need to develop alternative sources of bandwidth.

Pointing out that the average American spends roughly the same amount of money for bandwidth of some form or another, whether on their cellphones, land-line phones, cable TV, broadband Internet, etc., as they do on heating, cooling, electricity, and gasoline, Wu makes the case that we need the data equivalent of the compost garden, rooftop solar panel, and electric scooter.

The difference, of course, is that while rooftop solar panels have recently started to be affordable on a per-wattage basis, Wu’s suggestion of “running your own fiber to your home” isn’t. Even if you did put down your own last-mile fiber, you still need some place to connect it to… the whole point is that the value of a network is equal to the number of end-points on the network, squared.

The other solutions Wu suggests: re-allocation of the EM spectrum, municipal fiber as a public utility, and increasing the amount of competition are all more practical ideas, but they require significant changes in governmental policy – the same government elected from a two party system where the two parties are both heavily influenced by broadband providers and telecommunications companies.

But Wu is not wrong in identifying a significant problem – especially since e-mail, telecommuting, and teleconferencing are all considered important ways to reduce energy expenditure. It is disheartening to believe that we’re simply switching from one artificially overpriced commodity to another.

Damn. Well, I guess there’s no way around it then. Looks like we’ll have to recreate the Internet from scratch by hooking up everyone to self-installed network connections. So what do you say, you lay the fiber if I use the backhoe?

Earlier today, Network World hosted a live chat with Jim Metzler, of Ashton, Metzler, and Associates on implementing WAN acceleration. They’ll have the full transcript up tomorrow on their site, but I attended and wanted to bring back a few highlights. 

-----

Question: What is the one question that should be asked when evaluating products that you don't see people asking?

Jim Metzler: One thing that I think that people should ask: What has gone wrong in previous deployments?  We have all been around long enough to know that things do tend to go wrong at least occasionally.  For example, some people have found that once they deploy a WAN optimization controller (WOC,) that they lose management visibility.

Question: We are constantly battling latency across our MPLS network.  We have retail stores that connect to HQ data center.  How do we improve WAN performance?  Do we need to implement QoS?  Or should we use a different WAN protocol for our Cisco routers?

Jim: MPLS comes along with service classes that promises a guaranteed latency.  For example, a given service class may promise that latency will not exceed 50ms.  If your problem is that you are not getting what you were promised, that is an issue to take up with your vendor, or, based on your contract, to possibly change vendors.  If the issue is that the latency that you are promised is not good enough, I need to know more about what the problem is.  For example, if the issue is that you are running chatty protocols over the WAN, then a WAN optimization appliance might be helpful.

Question:  How do WAN optimization technologies fit into a more VIDO oriented desktop/branch and/or more XML oriented apps?

Jim: The movement to implement virtual desktops is a bit behind the movement to deploy virtual servers.  As we deploy more virtual desktops, that will mean more traffic from the data center to the branch which will most likely need optimization.

Question: How does WAN accellerationfit into scenarios with heavy ICA traffic or Notes replication traffic?

Jim: WAN acceleration is a very broad topic.  Some applications (CIFS traffic that results from server consolidation) scream out for optimization.  Other traffic (VoIP) requires QoS so that other traffic (bulk file transfers do not interfere with it.

The bottom line is that there are differing traffic types and they often require different techniques.

Question: Is there any company that has the best all-around solution?

Jim: No.Okay, my answer was intentionally brief.  Your question goes to one of the key challenges facing IT organizations today.  A given supplier might have a great solution for data replication but not so great for CIFS.  Another vendor may have a great solution for CIFS, but not so much for data replication.

This presents IT organizations with a challenge - what problems are they trying to solve today?  Next year? You have to decide.  Do you choose the best solution to today's problem knowing it might be sub-optimal for tomorrow's?

The good news here is that over time the differences between the suppliers on common functionality (compression, caching, protocol acceleration) will diminish.

Question: What question do you hear the most about this product category?

Jim: “How do I get started with evaluating these products?”  “What new directions are the gear vendors taking these products?”

With regards to WOCs: Adding support for specific applications such as share point or SAP.  Creating templates to make it easier for IT organizations to configure the device to support key applications.  Embracing virtualization.

With regards to Application Delivery Controllers (ADCs): Offering virtualized solutions, adding security functionality, adding functionality such as XML processing, Integrating with Business Intelligence tools. 

Question: Do you see distinct advantages to implementing QoS in a WOC rather than the router level? 

Jim: This is a really fundamental question.  I believe that one of the reasons that we have not implemented WOCs more broadly is that we have not answered some basic questions such as what functionality should be done where.

The question is further blurred by the fact that WOCs are being integrated into routers making it tough to say where QoS was implemented.  Bottom line: I think you can make either approach work.  It comes down to factors such as how rich is the QoS functionality in the WOC and how easy is it to configure and manage the QoS functionality in the WOC or the router.

Question: Some users claim that acceleration claims made by the vendors are bogus, that claims of 400% improvements are marketing garbage (as you can’t improve speed faster than the original base speed) What are your thoughts on speed claims by vendors?

Jim: The acceleration claims made by the vendors represent a test done in a laboratory.  While these might give some insight into how the devices will perform in production networks, they are not definitive.  IT organizations must test the devices in their network to understand what types of improvements they will realize. 

Question: Any thought on masking poor application architectures with WOC or ADC products?

Jim: This is a hot button to me!

For example, most IT organizations do not spend much attention on how apps will perform over the WAN during development or acquisitions.

A lot of our current problems would go away if apps were designed to run better over the WAN.

For example, one company I worked with found out that the browser in the branch offices were downloading a 3MB file just to open it up and extract a 10 digit ID.  Talk about a badly designed application!

Question: Where is the next big thing: security, storage, or--?

Jim: I think that there will be a lot of next big things.  I recently wore a series of articles for Network World in which I described "The Perfect Storm." 

Let me explain.  Today over half of the outages occur based on ineffective change or configuration management.  once we move to an environment with a virtualized desktop, routers running VMs running all kinds of things, communicating with a virtualized application delivery controller (ADC) that front ends a Web server, app server and database server, all of which are virtualized and which use virtualized storage, then the situation gets very thorny. 

Can you imagine how much many more outages will occur in a fully virtualized environment?

Sticking with this: Today when an application is not performing well it is difficult to identify the root cause.  That will be much more difficult in a fully virtualized environment. 

SOA really worries me.  By SOA, I mean Services oriented architecture.  Some think SOA is the precursor to an SOB and they may be right.

With an SOA, an application is comprised of multiple web services - for the sake of example - say 8.  Now, these Web services are running in different data centers - say 5.

Now the WAN impacts that application performance many more times than it does in today's n-tier applications.  This will be a huge challenge, but it gets worse - these Web services are reusable.

That means that multiple apps are using the same web services at the same time - that drives the need for QoS for Web services

This is all extremely demanding. 

Then there is Web 2.0 and mashups. With a Mashup your app is using apps designed and managed by other entities.  You have no control or visibility into those other apps.  This will be extremely challenging.