I can’t believe I missed this the first time around.

I was so focused on how the online Olympic video was getting through the last mile, that I completely forgot to ask: How the heck are they getting it from Beijing to the U.S.?

Douglas Gourlay at Cisco has been blogging about how NBC’s been using Cisco’s Wide Area Application Services (WAAS) for WAN optimization, so that NBC’s video editors can use three 155Mbps OC-3 pipes, combined and load-balanced (with, of course, Cisco gear) to get the files directly from Beijing. While I’m not 100% sure on “as if they were stored locally,” holds true, it’s clear that WAAS is capable of some amazing stuff – we know because NetQoS has SuperAgent integration on WAAS devices and ACE load balancers. We track stuff like that all the time.

“This reduces operating costs of housing, air travel, transportation, and food. Avoiding 800 airplane trips also supports NBC’s green initiatives for the Olympic Games.”

It also probably makes the video editors a bit grumpy that they didn’t get to go to Beijing.

What I’m curious about is what will happen after the Olympics. Just as Olympic stadiums still stand – and are used – in every host city, I’m wondering if the infrastructure that NBC has to Beijing to deliver high definition video will remain after the Olympics. As China starts to become a new superpower, more news and information is bound to come from Beijing, after all.

And if this can be done for one series of events in one major city, is it that far off from having video-heavy WANs in every city to cover every major event?

While we’ve talked quite a bit about what impact the Olympics may have on an enterprise network’s performance, we haven’t talked much about the performance of the NBC site hosting the live streaming of the Olympics. 

According to Jason Perlow at ZDNet, Limelight networks (which hosts the streaming videos) deployed the videos by going to the public internet by hosting the content more locally – at the ISP.  That means you’re viewing the Olympics through your ISP’s internal network, and the broader internet doesn’t even enter into the connection. 

This is smart thinking, it appears to be working, and by all measures this should be applauded.  Perhaps even duplicated – if you know that multiple employees will download the same content, local hosting on the LAN is preferable to duplicate download streams tying up the more expensive, slower WAN lines.

From the enterprise end of the equation, the fact that Limelight is delivering Olympics video more effectively just means that IT managers cannot count on their servers going down from being unable to handle the demand – IT managers still need to monitor their own networks for performance problems when a big event like the Olympics come up. 

However, it would be wrong to assume that Limelight’s strategy is the only reason why Olympic live-streaming hasn’t slowed to a trickle.

First of all, the site blocks 95.44% of visitors from accessing the content – because it limits the content only to those in the United States.  That’s a lot of people.

Secondly, the site requires Microsoft Silverlight. Most people don’t have Silverlight installed.  Some can’t even install it on their systems.  And there are certainly going to be a quite a few people who just didn’t think installing Silverlight was worth the bother to watch five minutes of Olympic footage they may be mildly interested in. 

And finally – none of the really popular sports are being streamed.  Gymnastics, Women’s Beach Volleyball, Swimming (with the exception of synchronized) and most of the track and field events aren’t available live. So you’re left with judo, fencing, and the decathlon.

So while it is a true technological wonder that the lights have stayed on and the site performs admirably – it is important to recognize that Limelight has not found a magic bullet to deal with extremely high internet video demand. 

Jim Metzler and Steve Taylor have another insightful article up at Network World – this one on the effects of the FCC ruling against Comcast over their BitTorrent blocking using deep packet inspection technologies.

Perhaps some of the best news for corporate network managers is that this is proof that equipment designed for DPI actually works – and evidently works well. So if you decide that you need more control for P2P traffic on your corporate network, this seems to be quite an endorsement. And, again pointing out that we’re not lawyers, there seems to us to be a fundamental difference in the ability to filter traffic on your own corporate network and on public networks.

This is true. But there is something here that is amiss.  Yes, deep packet inspection works.  Except for communication that is encrypted and decrypted.  Of course, if an employee is determined to jump through the hoops required to get a decent encryption on BitTorrent content, so that he can download the large, copyrighted, non-work related files at the work computer… well, at that point it ceases to be an IT problem and becomes an HR problem.

Chances are, anyone smart enough to set up encryption would be able to figure out to use their home broadband connections.  Can’t wait to get home? Set up a remote login. 

What is of more significance is the idea that it looks like at least some degree of network neutrality will be enforced by the FCC; the Comcast case may set a precedent and other ISPs may be wary of implementing DPI.  That may make a huge impact on deciding whether you want to try putting apps out on the cloud or whether you want to develop a solution on the WAN. 

A number of East Coast based customers of World of Warcraft have been experiencing connection delays and uncomfortable lag – and no one seems to know exactly where the problem is.

The New York Post says that Blizzard is blaming Time Warner Cable is for the problem:

"The only commonality between all the players experiencing these disconnects and extreme latency is Time Warner/Road Runner," the company said in a June 23 support post.

But the Digital Communications Director for Time Warner has said that the lags and disconnections are not on their end and points to the traceroutes as evidence.

Take a look at some of the traceroutes posted to the thread in question ... starting here, at comment #446: http://tinyurl.com/5gqe27

If you follow the commenter's posted trace results, you'll notice that it's only on TWC's Roadrunner (rr) network for the first 6 hops — with maximum response times of 10 ms. The response time jumps drastically at hop # 11 — when the trace is no longer on the Roadrunner network.

Scroll down further on the same page to comment #456, and you'll see something similar — a giant leap in lag times. However, this trace never touches our network. It starts at Verizon, goes to Alter.net at hop #5, and then jumps to ATT.net's network at hop #8. Hop #9 shows a response time of 114 ms — quite a jump from the 49ms at hop #8.

So, what’s going on?

One of the theories is that Time Warner is lying and is throttling World of Warcraft traffic, considering all the bad blood between savvy broadband users and major ISPs over BitTorrent throttling. And while I can’t prove that they’re not doing so, I have to admit that the theory doesn’t seem very likely because of the nature of World of Warcraft.

See, MMORPGs care more about latency than bandwidth. While patch downloads can be huge, the majority of the content of WoW requires low latency to provide instant responses to actions. Latency, in WoW can result in an annoyingly choppy game, and a multi-hundred millisecond delay may be the difference between slain dragon or hobbit pâté.

So from a bandwidth-saving perspective, a ISP wouldn’t have a whole lot of motive in blocking World of Warcraft or other MMORPGs.

Additionally, Comcast, Time Warner, and other cable companies were rumored to use BitTorrent throttling because both legal and copyright infringed video files competed with the standard television cable offerings of those companies. This also doesn’t seem to be the case – as while more generally, time spent playing WoW is time not spent watching TV, it’s not a specific competition. Indeed, MMORPGs are one of the key drivers for broadband speeds in the U.S., and I have trouble believing that TW or any other company would knowingly interfere with such a cash cow.

Indeed, I believe that TW might be reaching out to users to find out more about the problem because TW might be interested in solving the problem instead of losing customers to other ISPs like Verizon FIOS.

Of course, I don’t know anything – and I wish that I had some inside information to figure out what was going on and solve the problem. Not only would I look like a genius but every one of my friends who plays World of Warcraft would hoist me on their shoulders, and treat me like a Lich King for a Day. Sadly, I think that it’s going to take Blizzard and TWC together to try to triangulate why this problem is happening.

I’ve been getting a number of e-mails and comments asking why I haven’t yet written anything about the Russian/Georgian war and the supposed “cyber-warfare” taking place. ZDNet has written extensively about the DDoS attacks being waged against Georgian government sites.

At first, I thought that this was solely a security issue. As a general rule, I don’t like to talk a whole lot about computer security on Network Performance Daily because I lack the proper mindset to get around security – security experts are people who look at things and see how to break them down, network performance experts are generally people who look at things and see how to build them better. Besides, there are tons of blogs out there about computer security, and very few about network performance.

I’m not going to get into the geopolitical aspects of it, except to say that getting involved in a land war in Asia is one of the “classic blunders.”

However, I did start thinking about things… I mean… wasn’t the Internet partially designed to be a resistant form of communication in case the Russians ever attacked? The irony of the Russians effectively taking down a country’s Internet is… well, it’d be funny if it wasn’t for all the people dying.

What this does tell me, however, is that cloud computing (and I’ll continue to call it that despite Dell’s claim to the term,) has a long way to go. While the Internet can be cheaper and simpler than having a fully-fledged IT department monitoring in-house servers and applications on leased lines over a WAN, the one problem that in-house IT has licked is fault.

For the most part, we’ve managed to get it so that we no longer worry about fault on the enterprise network. It was a while ago that we passed the 99.999% uptime mark. So while we may worry about security and performance, we typically don’t have to worry about the network not working.

But cloud computing still has fault problems. And it doesn’t take the Russians attacking. I love Stumbleupon, but they went down for a few minutes yesterday – Twitter also, but they’ve got problems. Even Gmail, which I greatly rely upon for my personal e-mail, went down for a little while earlier this week.

By and large, cloud computing makes great solutions for smaller companies and start-ups because of the low cost, low maintenance, and portability. However, the tradeoff is reliability – Internet applications simply aren’t as reliable as the bulky solutions that get things done when a single hour of downtime can mean thousands in lost business.

There really is no such thing as a private cloud. The entire concept revolves around using IT services offered from outside companies, which connect on public lines through to shared servers.

This is not to say that there is no room for the cloud in enterprise computing but that incidents like the South Ossetian war show that Internet applications suffer from one fatal flaw: They’re on the Internet.

One of the things that we hear most often is that it’s hard to identify the problem being the application, server, or network.  We can now officially add to this list: Application, server, network, or virtualization layer. 

Virtualization is to IT sorta what light is to physics.  Is it a wave?  A particle?  Is it a software thing?  Is it hardware?  Virtualization is… virtualization.  It’s something different, something new, and something which is very very difficult.  Also like physics.

And when things go wrong with virtualization, they can go wrong very very badly. According to Network World:

Many VMware customers Tuesday were prevented from logging onto their virtual servers as a bug distributed in a software update effectively stopped the boxes from powering up….

"This certainly appears to be the most publicized bug for VMware so far, and I think it is damaging to VMware and virtualization as a whole. The hypervisor is the lowest software level on the server and if you have an issue like this, boom, all your infrastructure is down," says Gary Chen, a senior analyst with Yankee Group. "Software will always have bugs, but a widespread issue like this that affects all VMs is really damaging, especially at this point in time where virtualization is starting to take off. VMware is going to have to fix this fast, provide an explanation, and outline what they will do to strengthen their QA in the future."

The bug is preventing users from powering on virtual machines, though VMs already running continue to do so.  There is a workaround – set the date back, turn on the VM, reset the date – but that can be tricky when certain applications need accurate timing information  – banking records, for instance.

Of course the fact that it’s complicated means that it’s even harder for lay people to understand. I’ll bet you that even though the real problem is the virtualization layer, end-users who find this problem prevents them from working blame the network.

by Brian Boyko
Editor, Network Performance Daily

USA Today publishes “CyberSpeak” from columnist and radio talk-show host (not to be confused with “talk radio show host”), Kim Komando. For over a decade now she has been helping people become more comfortable with digital technology and the Internet. She has won the 2007 Gracie Award, and is a journalist I greatly admire.

I give her that introduction, because I’m going to rip her latest USA Today column, entitled “Web Delivers New Worry for Parents: Digital Drugs,” to shreds, turn the shreds into mulch, and turn the mulch into compost.

We all know that music can alter your mood. Sad songs can make you cry. Upbeat songs may give you an energy boost. But can music create the same effects as illegal drugs?

This seems like a ridiculous question. But websites are targeting your children with so-called digital drugs. These are audio files designed to induce drug-like effects.

All your child needs is a music player and headphones.

The article goes on from seizing the “maternal fear gland” by the throat to explain that she’s talking about binaural beats, which supposedly affect your brain waves and give the listener a high not unlike taking a drug. If this sounds familiar, it’s a lot like the plot behind the William Shatner-created “Tekwar” series of novels.

(Continued...)

Continue reading "Won’t somebody think (better) of the children?" »

A study commissioned by VoIP provider Inclarity and conducted by YouGov among workers in the United Kingdom found that up to 60% of those polled had, during the previous year, had experienced a full day’s disruption of their company’s phone system. And according to SC Magazine UK, telephone downtime costs UK businesses around £14,000 each day, or, with the current exchange rate, about three billion U.S. dollars per minute.

Worse still, 61% of respondents didn’t have, or weren’t aware of, a disaster recovery or business continuity plan in the event of phone service problems. Considering that kind of money, it is extremely important to have a backup plan for the phones, be able to isolate performance issues, and speed recovery times. Because the only thing worse than one day without phone service is multiple days without phone service. Or baby cheetah murder. I suppose baby cheetah murder is worse.

VoIP is sensitive to latency and jitter, and anything that interferes with either latency or jitter will create bottom-line impacting problems. Now think about all the possible things that can increase latency or cause jitter either within your network or outside of your control. Sobering thoughts, huh?

Monitoring VoIP performance and being warned before problems impact the bottom line is very important, but sometimes, things like hurricanes, tornadoes, backhoes, and little kids pushing the bright shiny red button will happen. it always helps to have a backup plan in place of some sort of catastrophic failure.

IT World has published a fascinating look at the “Nerdlympics” – nerdy alternatives to the blah in Beijing. A neat read, it looks at the world of speedcabling, obfuscated code contests, Soldering, Buzzword Bingo, and Speedrunning Super Mario Brothers.

There are of course, a few things they left out.

For example, there’s a documentary about CalTech’s Basketball team, called “Quantum Hoops.” It details the team’s goal to actually win a single game, despite the fact that the team has more valedictorians than high school varsity basketball players.

Of course, CalTech’s Basketball team is WAY less nerdy than Princeton and Middlebury’s Quiddich league.

“The role of the snitch is filled, not by an enchanted, plum-sized, golden-feathered ball, but by a hyperactive college student dressed in yellow with a penchant for running and wrestling. To capture the snitch, one of the seekers must grab a black sock hanging from the snitch’s shorts.”

I have found it impossible not to laugh at that previous sentence.

Of course, ultimately, there’s not much difference between Geeks and Sports Geeks…

The Wall Street Journal has an article out about “Why the Olympics Scare Tech Pros.”  But really, should this even be scary anymore? 

We’ve known for quite some time that major cultural events, such as the Olympics, can increase recreational traffic on the network as people tune in to catch sporting events.   These events can generate enough traffic to push many enterprise networks to the limits and adversely affect business-critical application performance.  And NBC plans to stream footage of the Beijing games over the Internet.

There are a number of solutions including QoS policies, limiting bandwidth to certain subnets… I particularly like the approach that Brunswick (the bowling guys) are taking.

[Cathy] McClain [divisional chief information officer at Brunswick] can’t just block streaming videos. Some Brunswick employees, the marketing department for example, have to watch the Olympics for work reasons. And blocking sites doesn’t fit with the company culture. Instead, she’s letting workers do whatever they want. But if the network becomes strained, a message will pop up on employees’ computers asking whether they’re watching the video for work-related reasons, and if not, could they please wait until off-peak hours.

The messages explain that Brunswick is trying to save money and McClain includes her phone number so that anyone who has a question can call for an explanation. And they don’t block the video – they just ask workers if they have to watch right now.

It’s a backlash-free way to protect the network. “My community is polite,” McClain tells us. “They get it.”

So, yes, there needs to be policies in place for this sort of thing. But it’s not like this is any sort of big surprise.  We’ve had four years to prepare for this.  Four.  Years.  And chances are if you’re reading this you know about what streaming video can do to your network if left unchecked, you’ve probably lived through a few March Madnesses and Super Bowls and World Cup and World Series and Shriner Bowls…

Besides, the Olympics are crap.

What?  They are! 

First, and to the chagrin of those guys at Brunswick, there are no bowling events.  They just completely ignore the sport.  How can you even take the Olympics seriously if they don’t include bowling?  We’re talking about a franchise whose winter version has included curling.  Curling is practically the same thing, only colder and with brooms. 

Secondly, the International Baseball Federation (IBAF) is changing the rules of baseball at the last minute.  You can’t do that!  You can’t really even call it baseball if you change the rules.  Call it… I don’t know.  Whinyball. 

And of course there’s the whole China/human rights thing. 

Worst of all, the Olympic games in Beijing is pretty much dominated by sports.  Seriously, someone should talk to their marketing department.  I feel pretty confident based on informal polling of myself and my friends at the Linux User Group, the guys at the comic book store, and my LARP buddies – and they pretty much agree that the Olympics has to have some sort of draw other than sports, because really, who likes watching that stuff?