I can’t believe I missed this the first time around.

I was so focused on how the online Olympic video was getting through the last mile, that I completely forgot to ask: How the heck are they getting it from Beijing to the U.S.?

Douglas Gourlay at Cisco has been blogging about how NBC’s been using Cisco’s Wide Area Application Services (WAAS) for WAN optimization, so that NBC’s video editors can use three 155Mbps OC-3 pipes, combined and load-balanced (with, of course, Cisco gear) to get the files directly from Beijing. While I’m not 100% sure on “as if they were stored locally,” holds true, it’s clear that WAAS is capable of some amazing stuff – we know because NetQoS has SuperAgent integration on WAAS devices and ACE load balancers. We track stuff like that all the time.

“This reduces operating costs of housing, air travel, transportation, and food. Avoiding 800 airplane trips also supports NBC’s green initiatives for the Olympic Games.”

It also probably makes the video editors a bit grumpy that they didn’t get to go to Beijing.

What I’m curious about is what will happen after the Olympics. Just as Olympic stadiums still stand – and are used – in every host city, I’m wondering if the infrastructure that NBC has to Beijing to deliver high definition video will remain after the Olympics. As China starts to become a new superpower, more news and information is bound to come from Beijing, after all.

And if this can be done for one series of events in one major city, is it that far off from having video-heavy WANs in every city to cover every major event?

While we’ve talked quite a bit about what impact the Olympics may have on an enterprise network’s performance, we haven’t talked much about the performance of the NBC site hosting the live streaming of the Olympics. 

According to Jason Perlow at ZDNet, Limelight networks (which hosts the streaming videos) deployed the videos by going to the public internet by hosting the content more locally – at the ISP.  That means you’re viewing the Olympics through your ISP’s internal network, and the broader internet doesn’t even enter into the connection. 

This is smart thinking, it appears to be working, and by all measures this should be applauded.  Perhaps even duplicated – if you know that multiple employees will download the same content, local hosting on the LAN is preferable to duplicate download streams tying up the more expensive, slower WAN lines.

From the enterprise end of the equation, the fact that Limelight is delivering Olympics video more effectively just means that IT managers cannot count on their servers going down from being unable to handle the demand – IT managers still need to monitor their own networks for performance problems when a big event like the Olympics come up. 

However, it would be wrong to assume that Limelight’s strategy is the only reason why Olympic live-streaming hasn’t slowed to a trickle.

First of all, the site blocks 95.44% of visitors from accessing the content – because it limits the content only to those in the United States.  That’s a lot of people.

Secondly, the site requires Microsoft Silverlight. Most people don’t have Silverlight installed.  Some can’t even install it on their systems.  And there are certainly going to be a quite a few people who just didn’t think installing Silverlight was worth the bother to watch five minutes of Olympic footage they may be mildly interested in. 

And finally – none of the really popular sports are being streamed.  Gymnastics, Women’s Beach Volleyball, Swimming (with the exception of synchronized) and most of the track and field events aren’t available live. So you’re left with judo, fencing, and the decathlon.

So while it is a true technological wonder that the lights have stayed on and the site performs admirably – it is important to recognize that Limelight has not found a magic bullet to deal with extremely high internet video demand. 

by Brian Boyko
Editor, Network Performance Daily

USA Today publishes “CyberSpeak” from columnist and radio talk-show host (not to be confused with “talk radio show host”), Kim Komando. For over a decade now she has been helping people become more comfortable with digital technology and the Internet. She has won the 2007 Gracie Award, and is a journalist I greatly admire.

I give her that introduction, because I’m going to rip her latest USA Today column, entitled “Web Delivers New Worry for Parents: Digital Drugs,” to shreds, turn the shreds into mulch, and turn the mulch into compost.

We all know that music can alter your mood. Sad songs can make you cry. Upbeat songs may give you an energy boost. But can music create the same effects as illegal drugs?

This seems like a ridiculous question. But websites are targeting your children with so-called digital drugs. These are audio files designed to induce drug-like effects.

All your child needs is a music player and headphones.

The article goes on from seizing the “maternal fear gland” by the throat to explain that she’s talking about binaural beats, which supposedly affect your brain waves and give the listener a high not unlike taking a drug. If this sounds familiar, it’s a lot like the plot behind the William Shatner-created “Tekwar” series of novels.

(Continued...)

Continue reading "Won’t somebody think (better) of the children?" »

The Wall Street Journal has an article out about “Why the Olympics Scare Tech Pros.”  But really, should this even be scary anymore? 

We’ve known for quite some time that major cultural events, such as the Olympics, can increase recreational traffic on the network as people tune in to catch sporting events.   These events can generate enough traffic to push many enterprise networks to the limits and adversely affect business-critical application performance.  And NBC plans to stream footage of the Beijing games over the Internet.

There are a number of solutions including QoS policies, limiting bandwidth to certain subnets… I particularly like the approach that Brunswick (the bowling guys) are taking.

[Cathy] McClain [divisional chief information officer at Brunswick] can’t just block streaming videos. Some Brunswick employees, the marketing department for example, have to watch the Olympics for work reasons. And blocking sites doesn’t fit with the company culture. Instead, she’s letting workers do whatever they want. But if the network becomes strained, a message will pop up on employees’ computers asking whether they’re watching the video for work-related reasons, and if not, could they please wait until off-peak hours.

The messages explain that Brunswick is trying to save money and McClain includes her phone number so that anyone who has a question can call for an explanation. And they don’t block the video – they just ask workers if they have to watch right now.

It’s a backlash-free way to protect the network. “My community is polite,” McClain tells us. “They get it.”

So, yes, there needs to be policies in place for this sort of thing. But it’s not like this is any sort of big surprise.  We’ve had four years to prepare for this.  Four.  Years.  And chances are if you’re reading this you know about what streaming video can do to your network if left unchecked, you’ve probably lived through a few March Madnesses and Super Bowls and World Cup and World Series and Shriner Bowls…

Besides, the Olympics are crap.

What?  They are! 

First, and to the chagrin of those guys at Brunswick, there are no bowling events.  They just completely ignore the sport.  How can you even take the Olympics seriously if they don’t include bowling?  We’re talking about a franchise whose winter version has included curling.  Curling is practically the same thing, only colder and with brooms. 

Secondly, the International Baseball Federation (IBAF) is changing the rules of baseball at the last minute.  You can’t do that!  You can’t really even call it baseball if you change the rules.  Call it… I don’t know.  Whinyball. 

And of course there’s the whole China/human rights thing. 

Worst of all, the Olympic games in Beijing is pretty much dominated by sports.  Seriously, someone should talk to their marketing department.  I feel pretty confident based on informal polling of myself and my friends at the Linux User Group, the guys at the comic book store, and my LARP buddies – and they pretty much agree that the Olympics has to have some sort of draw other than sports, because really, who likes watching that stuff? 

Recently, we did an unscientific (and really, I cannot stress that word enough) but real-world test of performance using the Ono plugin for BitTorrent client Vuze/Azureus. Our results were inconclusive.

David Choffnes, the author of the Ono plugin, responded to the test in our comments section of that article:

Regarding your results, it is difficult to run controlled experiments because even when you download the same torrent, you're doing it at different times with necessarily different swarms. My research group's evaluation is not limited by this, and we showed that performance improves *on average*.

Also note that if Ono doesn't find any nearby peers, it can't help your performance. You can see if Ono found nearby peers (and is using them) in the "Ono" plugin view … Also, the plugin's effectiveness is limited by the fact that "only" 180,000 users have installed Ono. The more people use it, the more likely you'll find nearby peers.

One last point -- even when Ono doesn't dramatically improve performance, it encourages better "Internet citizen" behavior. Why transfer data from halfway across the world when you can get the same data and (potentially better) performance from peers nearby? Ono makes it easier to do the latter, which should eventually help everyone using the Internet.

Ono is part of Aqualab, a Northwestern University computer science project researching large-scale distributed computing. Choffnes, a doctoral candidate, will present his findings at SIGCOMM next month, and his paper on the subject can be found here – which is great if you like trigonometric functions in your technical literature.

There’s also a telling paragraph which may explain why we got the results we did for our tests (other than just the random variability of different BitTorrent swarms), instead of a massive throughput boost.

In our analysis, we compare statistics from peers located by Ono (referred to as Ono-recommended peers) to those from all peers selected at random by the BitTorrent protocol, which also includes those located by Ono.

In Network Performance Daily's analysis, we compared statistics from peers located by Ono combined with peers selected at random from the BitTorrent protocol, against only peers selected at random from the BitTorrent protocol.

To determine the cosine similarity value for a peer, Ono must be able to compare its ratio maps with those of other peers. The latter information can be obtained in a number of ways: through direct exchange between peers, from distributed storage and from trackers. Ono currently supports the first two options. With direct exchange, when two peers running the Ono plugin perform their connection handshake, the peers swap ratio maps directly… Though Ono enjoys a large user base, it is still a small fraction of the total BitTorrent population. Thus Ono also attempts to perform DNS lookups on behalf of other peers that it encounters, to determine their ratio maps. This enables Ono to perform biased peer selection over a much larger set of peers, including those not running the Azureus client. From both direct exchange of ratio maps and DNS lookups, our Ono clients locate over 180, 000 peers per day using our CDN-based approach.

When Ono determines that a peer has similar redirection behavior, it attempts to bias traffic toward that peer by ensuring there is always a connection to it, which minimizes the time that the peer is choked. Due to limitations of the Azureus plugin API, we are currently unable to bias other aspects of peer connections, e.g., the bandwidth allocated to each connection.

In addition to Ono, Aqualab also does other projects that are designed for improving Internet performance in a number of other areas. Choffnes’s advisor, Dr. Fabian Bustamante, has been working on "sustainable scalability in distributed systems,” called the 3R project. Many P2P and internet VoIP systems are built in a way that routing is controlled at the application layer, and that in order to identify better paths and faster throughput, the application probes the network environment repeatedly, as the application has no quick way to determine whether a particular peer or node is performing well except by trying to connect to it. The 3R project seeks to decrease probing by re-using the view of the network gathered by long-running, ubiquitous services.

While enterprise networking and Internet networking are two different beasts, performance advances in one usually lead to advances in the other, and with cloud computing promising to make enterprise networking a hybrid of LAN, WAN, and Internet connectivity, these advances remain important.

Series Summary and Editorial
Part One: Interview with Texas State Rep. Joe Driver
Part Two: Interview with Matt Miller, Institute for Justice
Part Three: Interview with Capt. RenEarl Bowie, Texas Private Security Bureau

by Brian Boyko
Editor, Network Performance Daily

We’ve written three stories and conducted three interviews regarding HB2833.  The first was with the author of the law, Texas State Representative Joe Driver, the second with Matt Miller of the Institute for Justice, and the third with Texas Private Security Bureau Captain RenEarl Bowie.

Here is our editorial summary:

HB2833, the law designed to make changes to laws regarding private investigation but has PC and Network techs worried that their work may now be illegal, has caused confusion and worry from normal people doing normal jobs in a normal manner.  Whatever the original intent of the law, it is clear to see from its effects that the law itself is poorly written. 

Ultimately, words like “open to interpretation” and “case-by-case basis” are not words you want to use when describing either the meaning of, or enforcement of, the law.

So, where did things go wrong?  I think the man problem was a key misunderstood concept by Texas State Rep. Driver when he wrote the law.  It is clear from the interview with him that he believes that there is a clear and well defined line between “retrieval of data” and “investigation.”

“’Review, analyze, and investigate’ are the three key words, in my opinion, that drive the need for people to have some kind of license. Because if they're doing some of that, then they don't need to be - it doesn't need to be just anybody able to do that - they need to have somebody that has a security license. But if someone's just retrieving information and providing information for someone who is going to analyze, to use one of the words, then that's just a regular computer repair person.” – Rep. Driver.

But what Rep. Driver simply did not realize is that in the practical realities of IT, no such line exists. Any and every interaction that any IT person has with a computer requires some sort of “review, investigation and analysis,” whether it’s simple troubleshooting or complex network latency optimization. 

I can see where Rep. Driver was going with the law and what his intent was when writing it – rooting around through someone’s Windows Recycle Bin can be just as invasive as rooting around in somebody’s trash. 

But rooting around in the guts of a computer to discover the cause of a malware infection is different from rooting around in the guts of a computer to discover infidelity.  However, instead of making the criteria of “investigation” the purpose and use to which the information could be put, the law makes the criteria the way that the information is stored – “computer-based data not available to the public.”  The end result is that the net was cast too widely. 

Compounding this problem is the interpretation provided by the Texas State Private Security Bureau of the law – a literal one.

“Computer repair or support services should be aware that if they offer to perform investigative services… they must be licensed as investigators” – Texas Private Security Bureau Opinion Summary.

Unlike the law itself, the opinion summary is an unambiguous statement, and while Capt. Bowie may say that the law will be interpreted on a “case-by-case” basis, that is not what is in the official statement of opinion. 

As for the court case brought by the Institute for Justice – unfortunately, the Institute for Justice seems to want to fight this case on Constitutional grounds.  However, that will be a hard sell, as qualifications and licensing are clearly powers that states exercise, from state bar associations for lawyers, and state medical boards for doctors.  If the state of Texas wants to make a PI license a requirement for PC repair techs, it certainly has the power to do so.  It may be absurd, but absurdity is not unconstitutional. 

So, where does that leave technical practitioners like network engineers and PC repair gurus?  As a practical matter, I think most people are going to continue going about this, “business-as-usual” style and make a stink only after the law is enforced on some, most likely unsuspecting, tech somewhere in Texas.

The good news is that I think that it is indeed possible to clarify and change the law through the legislative process – Rep. Driver has stated that he would indeed make changes to the law if it needs clarification or amendment. 

It clearly does.  

By David Byrne,
General Manager of Managed Services

Our tools here at NetQoS deliver the capabilities and insight needed to understand what’s going on across the network from a performance perspective to improve application delivery. Some customers find it difficult to maximize the performance of their monitoring tools given the constant change they’re experiencing, which includes everything from new applications to infrastructure changes to staff turnover. And ironically, customers need to fully utilize their performance monitoring tools to minimize the risk of change and ensure performance remains optimal as they deploy new technologies and make alterations to the infrastructure.

While training helps improve knowledge and skills, a managed services approach can make more sense for customers undergoing these challenges.  For example, a company may go through a rapid amount of change in a short period and simply can’t keep up with the performance monitoring on its own.  Or, perhaps a company might have trouble recruiting and retaining technical employees.  Or maybe a company realizes the IT team’s core strength lies outside performance monitoring and doesn’t want their best engineers spending all their time on it.

For these reasons, we’re announcing the NetQoS Managed Services today, which means we can offer customers the benefit of our talent and resources to perform the traditional tasks required to maintain optimal network performance. Because we can deliver our expert knowledge and best practices at a reduced cost compared to a full-time equivalent, many companies will find our Managed Services a compelling option.

Think of all the devices going out on the network today.  Years ago, you didn’t have to worry about things like streaming audio or video or BlackBerry/iPhone compatibility. But change is constant, and right now it’s exploding, which makes putting the right technical resources on the right tasks critical.  For many companies, it doesn’t make sense to have their broadly skilled engineers focus only on one or two tools.   There is a value to focusing on performance, but for many companies, NetQoS Managed Services might be a better way to do it. 

By Kevin Davis
Adapted from “Sources of Latency” Whitepaper

When network users call the Help Desk to report poor application performance, you don’t typically hear things like “The router’s CPU is too busy!,” “The network utilization is above 70%!,” or “The carrier path has failed-over to a sub-optimal path.” Instead, what you’re likely to hear is “The network is slow” or “The calls on my IP phone sound terrible.”

Complaints that end-users lodge are nearly always based their quality of experience using the application. And their quality of experience is almost always reliant on time.

Anytime a significant delay occurs in the delivery of network data, application performance suffers. Depending on the type of application and how it works, variances in network delay can have a severe impact on application performance thereby degrading end-user’s experiences.

Two important measurements of time intervals in network transmission systems are referred to as “latency” and “jitter”. Understanding latency and jitter sources and how their values vary in network architectures is critical to engineering application performance and optimizing information resources. For many regular readers, this will be old-hat, but we’ll go over it again.

Network latency is the amount of time it takes for a packet to be transmitted end-to-end across a network and is composed of five variables:

Network Latency = (Distance Delay) + (Serialization Delay) + (Queue Delay) + (Forwarding Delay) + (Protocol Delay)

Serialization Delay refers to the amount of time it takes for a network interface (such as a router’s interface or computer’s NIC) to perform bitwise transmission of a frame unto the outbound media, Forwarding Delay is the amount of time it takes a network device to process a frame/packet by performing a destination address lookup and forwarding the frame/packet to the outbound interface, and Protocol Delay is the amount of time that access or transmission algorithms may contribute to the delay of a network frame, and is typically introduced at the endpoints of the data transmission system.

Serialization delay, on a per-packet basis, becomes insignificant at data rates above 1.544 Mbits/s – or a T1. Forwarding delay is typically insignificant in modern routers and switches (when appropriately configured – significant delay can occur in misconfigured routers.) And Protocol delay typically occurs at the access layer or the end points. So the two major variables that have the most effect on network latency are Distance Delay and Queue Delay.

Distance Delay is simply the minimum amount of time that it takes the electrical signals that represent bits to travel down the physical wire. Optical cable sends bits at about ~5.5 µs/km, copper cable sends it at ~5.606 µs/km, and satellite sends bits at ~3.3 µs/km. (There are a few additional microseconds of delay from amplifying repeaters in optical cable, but compared to distance, the delay is negligible.)

Distance delay can have a significant impact on application performance for applications that require a large number of network round trips in order to complete a transaction – for example, custom transactional based applications, database queries, and VoIP, which begins do degrade when one-way end-to-end latency exceeds 200-220 milliseconds.

One of the biggest sources of end-user ire are database queries designed to run over a LAN ported to the WAN. For example if a user executes a SQL database query that requests 100 rows of a database table, one row at a time, over a link with a latency due to distance of 60 ms, it would take approximately 6 seconds (60 ms * 100 turns) to complete the transaction. The same query executed by a user on a LAN connected to the same database server would take less than 2-3 ms to be completed, as the latency due to distance across the LAN is insignificant.

Queue Delay is the amount of time a packet must spend in a network buffer waiting its turn to be transmitted. Network interfaces transmit one frame at a time, typically one bit at a time. As such, when two or more packets are forwarded to a network interface at the same time, or close to the same time – one packet is transmitted while the others are put in a queue on the interface buffer to await their turn at the interface. Packets that are put into the queue must wait until they can be transmitted, adding milliseconds of delay.

Increases in Queue Delay can be measured and detected by monitoring traffic along a given network path. Typically, most intermittent increases in latency above the baseline distance latency can be attributed to network congestion. (In order to reduce the possibility of excessive queue delay, application servers that are members of the same application architecture should be placed on the same Ethernet switch and on the same VLAN to ensure they do not have to compete for uplink bandwidth when problems like the one pictured above occur.)

Worse still, if the problem gets worse and packets wait in increasingly longer lines within the queue, the buffer may become full and the packets may be dropped. Packet drop, in turn, causes TCP connections to throttle back on the rate of transmission.

Those are some of the main causes of latency – but what about jitter?

Jitter is a term that refers to the variance in the arrival rate of packets from the same data flow, and abnormal jitter values can negatively impact real-time applications like VoIP and video. Jitter is typically created by three different mechanisms in a network: variance in Serialization Delays due to variance in packet sizes, variance in per-packet Queue Delay due to packet spacing from multiple sources at a common outbound interface, or packets taking different routes from source to destination – perhaps due to per-packet load sharing or routing issues.

The most effective way to deal with jitter is by using low-latency queuing for VoIP and video traffic on network interfaces with large serialization and/or queue delays. In addition, endpoints (such as IP phones) can use jitter buffers or playout delay buffers in order to deliver received packets at a constant rate to the end consumer. These buffers are typically 30-50 ms in depth, and thus they attempt to manage jitter values within these values on any single one-way path. While these buffers technically add 30-50ms in latency, they significantly reduce jitter. Since human beings don’t start to notice latency in VoIP or VideoIP applications till it hits about 200ms, if latency can be kept to under 150 milliseconds, then jitter can be significantly reduced using this method.

by Patrick Ancipink

Things used to be easy.

No, wait.  Things never used to be easy.  In fact, they were horribly complex and frustrating to the point where engineers pull their hair out.  But now we usually expect around 99.99umpteen% uptime from our network equipment. 

So frustration today often stems from the new tasks that enterprise IT engineers are expected to handle beyond the routers and switches.  Application delivery controllers, WAN Optimization controllers, and more latency sensitive applications such as VoIP and Teleconferencing simply mean that the IT teams are being tasked with problems that require them to think in new ways about what it means to be in IT.

If you’ve been to any networking convention or conference, you’ve probably heard “in IT you either develop applications or deliver applications” more times than you’ve seen the Brady Bunch episode in which Marcia gets hit in the face with a football.  That’d doesn’t make it any less true. 

Ann Bednarz, writing for Network World, suggests that companies take research firm Gartner’s advice and look to hire “application delivery architects and engineers.” The idea is that there should be at least one person in the IT department whose full time job is worrying about application delivery and tuning on a WAN – someone who can converse with application developers and security teams and end users. 

At NetQoS, we’re trying to help companies get the information they need to either designate and train an existing member of the IT staff for these new responsibilities, or at least know what to look for when hiring for an Application Delivery Engineer position.

For example, some things we’re doing right now include our NetAnalyst training based on real-world examples on resolving complex network application issues, and integrating our multiple products together in the NetQoS Performance Center. 

But there are some more subtle ways in which we’re hoping to get this point across.  We argue that the most important metric for network performance management is application response time.  And while there’s many things that can affect application response time, the most basic is that your best possible application response time is limited by the latency of the connection (especially in financial applications,) multiplied by the number of connections that the application has to make.  Network engineers often focus on only one aspect of that formula, latency – while application developers only focus on the other aspect – the connections.  (That’s if they bother to think about the impact of the app on the network at all. And if they do, their test environment sorely lacks any similarity to the real world WAN.)  

So the value of developing the role of the Application Delivery Engineer, someone who can coordinate the two halves of that Application Response Time equation, becomes clear. 

by Brian Boyko
Editor, Network Performance Daily

Enterprises are seeing more adoption of 10 gigabit Ethernet according to a report by Network Instruments, and reported on their Network Observations blog that nearly one quarter of businesses are implementing 10G networks by the end of the year. The larger the company, the more likely a 10G rollout.

There’s certainly evidence of a trend, but is that evidence of a need-based demand? LAN technology at the gigabit Ethernet level typically has low latency – and I don’t see 10G Ethernet helping with that much if at all. Gigabit Ethernet is still a heck of a lot of bandwidth, especially compared to the bandwidth offered by WAN solutions. In any LAN/WAN/LAN traffic path, it’s almost always the WAN that proves to be the bottleneck.

But it is possible, with large VoIP networks, that you could be overloading the LAN capacity and decide to move to 10G for that reason. This could possibly explain why big companies are more likely to have 10G than smaller companies – because if you’re not hitting the bottleneck on the LAN, 10G doesn’t really help you deliver the applications any faster or effectively.

What I think is more likely is that 10G has hit a price point where it costs about as much to roll out 10G as it does the older technologies. Instead of 10G taking over the market from companies migrating from 1G, instead it seems that when companies choose to build new systems, they’re choosing to build them in 10G instead of 1G.

But again, it comes down to application delivery. And if we’re not delivering applications faster, the question is then asked – is there any application that is not feasible to execute on a 1G network for which a 10G network would be suitable?

Then I remembered that I’m a geek, and I like my toys.

Specifically, when I move into my new apartment next month, I’ll be back on my own router hardware. My current place has Ethernet built in – it’s a feature that saves me $50 a month, but the complex houses its own routers, which I have no capability to port-forward, which means that I can’t set up a remote desktop connection so that I can check on my home computer from work. And looking forward to being able to do that again reminds me that perhaps one of the new applications that could propel an adoption to 10G might be combining virtualization with remote desktop software – that is, making the end users work from their desk computers on a virtualized environment on a server. This means that you get more life out of older but still usable desktop hardware. According to the FAQ from RealVNC, at 100Mbps per connection, “most tasks will be indistinguishable performed remotely from if they were performed locally” Still, 100Mbps fills up a 1Gbps LAN pretty quickly. However, a 10Gb LAN might be able to accommodate this new application.

There are limitations – anything using full screen video or animation (a movie, or a 3-D environment) where there are rapid changes of every pixel will require even more bandwidth before it gets “choppy” – which will probably sink my plans of playing Half Life 2 on my Mac via a remote desktop connection to a PC. But this is certainly one of those “think about it” half baked ideas that may become reality in the near future.