By Kevin Davis
Adapted from “Sources of Latency” Whitepaper

When network users call the Help Desk to report poor application performance, you don’t typically hear things like “The router’s CPU is too busy!,” “The network utilization is above 70%!,” or “The carrier path has failed-over to a sub-optimal path.” Instead, what you’re likely to hear is “The network is slow” or “The calls on my IP phone sound terrible.”

Complaints that end-users lodge are nearly always based their quality of experience using the application. And their quality of experience is almost always reliant on time.

Anytime a significant delay occurs in the delivery of network data, application performance suffers. Depending on the type of application and how it works, variances in network delay can have a severe impact on application performance thereby degrading end-user’s experiences.

Two important measurements of time intervals in network transmission systems are referred to as “latency” and “jitter”. Understanding latency and jitter sources and how their values vary in network architectures is critical to engineering application performance and optimizing information resources. For many regular readers, this will be old-hat, but we’ll go over it again.

Network latency is the amount of time it takes for a packet to be transmitted end-to-end across a network and is composed of five variables:

Network Latency = (Distance Delay) + (Serialization Delay) + (Queue Delay) + (Forwarding Delay) + (Protocol Delay)

Serialization Delay refers to the amount of time it takes for a network interface (such as a router’s interface or computer’s NIC) to perform bitwise transmission of a frame unto the outbound media, Forwarding Delay is the amount of time it takes a network device to process a frame/packet by performing a destination address lookup and forwarding the frame/packet to the outbound interface, and Protocol Delay is the amount of time that access or transmission algorithms may contribute to the delay of a network frame, and is typically introduced at the endpoints of the data transmission system.

Serialization delay, on a per-packet basis, becomes insignificant at data rates above 1.544 Mbits/s – or a T1. Forwarding delay is typically insignificant in modern routers and switches (when appropriately configured – significant delay can occur in misconfigured routers.) And Protocol delay typically occurs at the access layer or the end points. So the two major variables that have the most effect on network latency are Distance Delay and Queue Delay.

Distance Delay is simply the minimum amount of time that it takes the electrical signals that represent bits to travel down the physical wire. Optical cable sends bits at about ~5.5 µs/km, copper cable sends it at ~5.606 µs/km, and satellite sends bits at ~3.3 µs/km. (There are a few additional microseconds of delay from amplifying repeaters in optical cable, but compared to distance, the delay is negligible.)

Distance delay can have a significant impact on application performance for applications that require a large number of network round trips in order to complete a transaction – for example, custom transactional based applications, database queries, and VoIP, which begins do degrade when one-way end-to-end latency exceeds 200-220 milliseconds.

One of the biggest sources of end-user ire are database queries designed to run over a LAN ported to the WAN. For example if a user executes a SQL database query that requests 100 rows of a database table, one row at a time, over a link with a latency due to distance of 60 ms, it would take approximately 6 seconds (60 ms * 100 turns) to complete the transaction. The same query executed by a user on a LAN connected to the same database server would take less than 2-3 ms to be completed, as the latency due to distance across the LAN is insignificant.

Queue Delay is the amount of time a packet must spend in a network buffer waiting its turn to be transmitted. Network interfaces transmit one frame at a time, typically one bit at a time. As such, when two or more packets are forwarded to a network interface at the same time, or close to the same time – one packet is transmitted while the others are put in a queue on the interface buffer to await their turn at the interface. Packets that are put into the queue must wait until they can be transmitted, adding milliseconds of delay.

Increases in Queue Delay can be measured and detected by monitoring traffic along a given network path. Typically, most intermittent increases in latency above the baseline distance latency can be attributed to network congestion. (In order to reduce the possibility of excessive queue delay, application servers that are members of the same application architecture should be placed on the same Ethernet switch and on the same VLAN to ensure they do not have to compete for uplink bandwidth when problems like the one pictured above occur.)

Worse still, if the problem gets worse and packets wait in increasingly longer lines within the queue, the buffer may become full and the packets may be dropped. Packet drop, in turn, causes TCP connections to throttle back on the rate of transmission.

Those are some of the main causes of latency – but what about jitter?

Jitter is a term that refers to the variance in the arrival rate of packets from the same data flow, and abnormal jitter values can negatively impact real-time applications like VoIP and video. Jitter is typically created by three different mechanisms in a network: variance in Serialization Delays due to variance in packet sizes, variance in per-packet Queue Delay due to packet spacing from multiple sources at a common outbound interface, or packets taking different routes from source to destination – perhaps due to per-packet load sharing or routing issues.

The most effective way to deal with jitter is by using low-latency queuing for VoIP and video traffic on network interfaces with large serialization and/or queue delays. In addition, endpoints (such as IP phones) can use jitter buffers or playout delay buffers in order to deliver received packets at a constant rate to the end consumer. These buffers are typically 30-50 ms in depth, and thus they attempt to manage jitter values within these values on any single one-way path. While these buffers technically add 30-50ms in latency, they significantly reduce jitter. Since human beings don’t start to notice latency in VoIP or VideoIP applications till it hits about 200ms, if latency can be kept to under 150 milliseconds, then jitter can be significantly reduced using this method.

by Patrick Ancipink

Things used to be easy.

No, wait.  Things never used to be easy.  In fact, they were horribly complex and frustrating to the point where engineers pull their hair out.  But now we usually expect around 99.99umpteen% uptime from our network equipment. 

So frustration today often stems from the new tasks that enterprise IT engineers are expected to handle beyond the routers and switches.  Application delivery controllers, WAN Optimization controllers, and more latency sensitive applications such as VoIP and Teleconferencing simply mean that the IT teams are being tasked with problems that require them to think in new ways about what it means to be in IT.

If you’ve been to any networking convention or conference, you’ve probably heard “in IT you either develop applications or deliver applications” more times than you’ve seen the Brady Bunch episode in which Marcia gets hit in the face with a football.  That’d doesn’t make it any less true. 

Ann Bednarz, writing for Network World, suggests that companies take research firm Gartner’s advice and look to hire “application delivery architects and engineers.” The idea is that there should be at least one person in the IT department whose full time job is worrying about application delivery and tuning on a WAN – someone who can converse with application developers and security teams and end users. 

At NetQoS, we’re trying to help companies get the information they need to either designate and train an existing member of the IT staff for these new responsibilities, or at least know what to look for when hiring for an Application Delivery Engineer position.

For example, some things we’re doing right now include our NetAnalyst training based on real-world examples on resolving complex network application issues, and integrating our multiple products together in the NetQoS Performance Center. 

But there are some more subtle ways in which we’re hoping to get this point across.  We argue that the most important metric for network performance management is application response time.  And while there’s many things that can affect application response time, the most basic is that your best possible application response time is limited by the latency of the connection (especially in financial applications,) multiplied by the number of connections that the application has to make.  Network engineers often focus on only one aspect of that formula, latency – while application developers only focus on the other aspect – the connections.  (That’s if they bother to think about the impact of the app on the network at all. And if they do, their test environment sorely lacks any similarity to the real world WAN.)  

So the value of developing the role of the Application Delivery Engineer, someone who can coordinate the two halves of that Application Response Time equation, becomes clear. 

In a few minutes, Jim Metzler of Ashton, Metzler, and Associates, will be delivering his keynote on the Next Generation NOC at NetQoS Symposium 2008 at Barton Creek Resort in Austin. Last week, we pre-recorded a podcast with Dr. Metzler regarding the speech he is about to give and what he means by a "next generation NOC."

He talks about the changing role of the NOC and moves in enterprises towards integrating what were once seperate stovepipe functions to focus on application delivery.

The podcast is below.

Today, in this podcast, we speak to Dr. Jim Metzler at Ashton, Metzler, and Associates regarding his handbook, "The Handbook of Application Delivery 2008" and his upcoming keynote speech a NetQoS Symposium 2008.

Kevin Davis, a senior consultant at NetQoS, will be presenting a few training sessions at Symposium about SuperAgent, the end-to-end response time module of the NetQoS Performance Center. This will include a training session about how to use time-based network metrics in troubleshooting.  He talks about his upcoming training session below.

In the session, I’m going to be covering the importance of using a time-based metric in troubleshooting, because end-users complain foremost about time.  For example, they’ll say “the application is running slow,” or they believe “the network is slow.”  To users, everything is based on time, that’s what they’re complaining about.  And they’re correct.

It’s very new to many people to think of performance in “time” although that may seem counterintuitive - because most people are used to reading utilization graphs.  With utilization graphs, however, we don’t know if 70 or 80 or 90 percent utilization is necessarily impacting the user experience.  I mean, we buy networking equipment, routers, switches, firewalls, servers, and we want them to be highly – or efficiently - utilized.  Seeing high utilization could indicate a problem – or it could just indicate that you haven’t over-purchased.  So you can have a link at 90% utilization or a router at ninety percent CPU utilization but you won’t know if that’s impacting the end-user without a time based metric.

It’s time-based data that tells you how the users are being impacted.  Sure, the utilization data – the interface utilization, memory utilization, I/O utilization, can often tell what is doing the impact.  But the time base shows you the degree of the impact – the real-world effect on end-users.  With a time-based instrument, such as NetQoS SuperAgent, you can find out where the delay increase is occurring, and whether it’s based in the network, server, or application. 

In fact, you can take a look at time-based data and make a determination very quickly as to which entity is creating the performance issue – the beautiful thing about SuperAgent, in particular, is that it trends by time 24/7, so not only can you determine how your important business applications are being impacted today, but you can go back and look at recurring patterns in performance issues.  You can see if today is worse than yesterday or last week or last month.

In the session, I’ll also be going over how to architect the data center for performance.  Placement of servers that participate in inter-architectures is critical for the health and performance of the application and indeed the data center.  We also talk about how different protocols, for example, Microsoft’s TCP/IP stack, can impact application performance by enhancing or degrading it. 

It’s important for servers that are serving the same application.  For example, a front-end Web server and a back-end Oracle database really should be on the same switch on the same VLAN.  That way they receive optimum service from the network.  If they do leave the switch, they’ll have to contend with bandwidth going up and down the switch links, and they’ll be switched and routed multiple times. 

Based on measurements from customer environments and from our own laboratories, when two servers are on different switches they can have up to 18 milliseconds delay between them.  If we think of that in the terms of network engineers of one millisecond per 100 miles, what in effect we’re doing when we put two different servers on different switches, or two different VLANs on the same switch, we’re making it look like those servers are 1800 miles apart – like one server is in Los Angeles and the other is in Memphis. 

by Patrick Ancipink
Director of Product Marketing, NetQoS

There’s been a lot of growth (and attendant hype) in technology areas like WAN optimization and application acceleration over the past few years, and for good reason. Anything that helps companies speed up and reduce the risk of strategic IT initiatives like consolidating data centers, turning up new branches or serving an increasingly mobile and scattered user community will be popular.

To help with cope with the increasing reliance on the WAN and keep latency in check, there are a dizzying array of vendors and products out there – but if you’re trying to determine precisely which techniques and technologies to implement for your specific needs, the array of vendors quickly goes from “dizzying” to “disorienting” and finally “nauseating.” 

Cisco’s been in this Tilt-a-Whirl™ of a market for a while (and NetQoS has been right there with them) and they’ve taken some big steps recently to provide a more holistic approach that centers on building an “application aware” network, rather than trying to highlight one type of implementation against another for a narrow set of capabilities.

NetQoS started working exclusively with Cisco closely to help customers evaluate, measure, and prove the effectiveness of WAN optimization and application acceleration deployments. As customers are moving from pilot phases into full production, the before/after measurements and comprehensive monitoring are critical to ensure customers are getting the benefits they intended and doing what they need to deliver application performance. 

To help get the word out, Cisco just launched a new section of their web site today that contains a wealth of information about, as they call it, “WAN and Application Optimization.” The downloadable presentation, Cisco WAN and Application Optimization Technical Overview Presentation, puts Cisco technologies (and complimentary ones, NetQoS included) into a useful context with a methodical approach and framework built around four steps: Profile and Baseline, Optimize, Evolve, and Operate. A whole Campbell’s Factory of Cisco alphabet soup technologies are included—WAAS, ACE, NBAR, Netflow, CBQoS, IP SLA, PfR—to show how they work in concert and what role they play in the bigger picture.

There’s also the Cisco WAN and Application Optimization Solution Guide , a very in-depth publication—like 227 pages deep—that is targeted for “technical personnel involved in the specification, design, and implementation of specific WAN and application optimization solutions.” We, here at NetQoS, are proud to have contributed several sections to book regarding the methodology and implementation of network performance monitoring for WAN optimization and application acceleration. 

(If you are looking for some lighter fare, the video on the site tells a nice story in about 6 minutes including an airshow, snowmobiles, windsurfers, and skydiving—interesting choices for demonstrating the criticality of serving video over the WAN.  Then again, some company somewhere has to make the recreational products, I suppose.)

For open source projects, v1.0 is generally a major milestone; one that is usually well earned.  After all, in open-source software, changes between versions are incremental and it can be a long time before hitting the 1.0 milestone.  For example, Mozilla – the original, before Firefox eclipsed it and it became SeaMonkey - spent four years as “beta” versions before finally getting the 1.0 designation – and it is notable that most of those beta versions were quite usable, Like much open-source software, it kept improving but just didn’t meet the developers exacting standards for a 1.0 release until it passed a threshold.

This 1.0 barrier was just reached for WireShark, the open-source packet sniffer. 

Is it a milestone?  Perhaps it’s just the ticking of the odometer over into the 1.0 area – the changes from the previous pre-1.0 version were minor – a new experimental version on MacOSX Intel, and some security related vulnerabilities patched.  However, WireShark remains a invaluable tool for anyone working in the network space.  Bill Alderson uses Wireshark for monitoring application performance, TCP behavior, retransmission symptoms, and protocol incongruities in situations in collaboration with NetQoS’s products – or where NetQoS’s products would be considered overkill.

Wireshark v1.0 is especially important because of its free-as-in-beer-ness.  Wireshark opened up network and application troubleshooting to rank-and-file IT staff – as anyone could download and use it, rather than having to wait for the network engineer to show up on-site or waiting for a third party.  Anything that helps the IT group solve problems faster is a good idea. 

Right now, Joel Trammell, our CEO, is at Sharkfest 08 (the first Sharkfest) at Foothill College, where in addition to Gerald Combs, the original developer of Wireshark, Dr. Vinton Cerf will be delivering a keynote. 

Additionally, with April Fools Day coming up tomorrow, you have to appreciate any program that allows you to play harmless practical jokes – and details exactly how on the official wiki. 

If there's one thing that we've been trying to impress, other than our abiding love for D&D, it is the idea that while a large amount of throughput is good, low latency should not be ignored. We've talked about how the idea of "chatty apps" that make lots of calls to the server for data can run slowly on a high-latency connection even when there is sufficient throughput.

But there are other instances where low latency is important as well. The biggest one by far is the financial sector.

A little more than a year ago, the New York Stock Exchange had a little glitch that caused a few seconds of non-responsive applications. The end result was that the NYSE had 3.3% loss in market value for that trading day - from a delay of mere seconds.

But even a delay of mere milliseconds can ruin a potential stock sale.

According to InformationWeek, electronic trading makes up 60 to 70 percent of the daily volume of the NYSE. Much like Olympic relay racers, a hundredth of a second is a quantifiable measure in the stock market. Trades are often queued by computer when some external factor applies - say that two brokers have both automatically set their computers to sell a stock called STOK to sell if it reaches $3.50 a share. A buyer comes along and offers to buy STOK at $3.50 a share. When the trade goes through, it's going to buy the shares from the broker - or more accurately, the broker's computer - that responded first.

But nearly all of the human elements have been taken out of the equation. Without those, the only differences between first place and second are a matter of network round trip time.

"A 1-millisecond advantage in trading applications can be worth $100 million a year to a major brokerage firm, by one estimate."

So, you can see why network performance isn't just about throughput. In the meantime, for your own network, you can check out our network latency calculator to find out what speeds you should theoretically be getting, and start working towards those speeds.

by Brian Boyko
Editor, Network Performance Daily

Apple hasn't made many overtures to business; the announcement that King Jobs had deigned to allow enterprise apps in his iPhone fiefdom took many by surprise. Apple's previous attempts at wooing enterprise customers, specifically the Xserve, seemed, in my opinion, more like a half-hearted reassurance to shareholders that they weren't completely ignoring the enterprise market.

Here's a telling point about Apple's attitude: Even though virtualization is one of the most important trends in enterprise computing, Apple makes the only operating system which cannot be run as a virtualized OS on generic hardware. It didn't allow Leopard Server to be run as a virtualized OS even on its own hardware until October 31st of last year.

While it's also true that Microsoft had - past tense - clauses in Vista EULAs which made it illegal to run Vista Home Basic and Vista Home Premium (but not Business, Enterprise, or Ultimate), as a virtual machine, those restrictions were eventually lifted in late January of 2008 - most importantly there was no deliberately imposed technical stumbling block that prevented those OS versions from being virtualized.

(One commentator, who I cannot remember, suggested that Microsoft's awkward anti-virtualization positioning was a move to prevent Apple from offering Parallels Desktop and a virtualized Vista pre-installed on new Apple consumer computers - but the ban has been lifted and Apple hasn't made any deal like that.)

Apple's MacOSX, on the other hand, checks to make sure it's running on Apple hardware, and will not run, otherwise. There are hacks to get around this, I'm sure, but they're much more difficult to pull off, may have stability problems - oh, and there's that whole "it's quite illegal" thing, too.

You can run MacOSX as a virtual server on an Xserve. But then it gets back to the application developers.

Enterprise application developers know today that they can pretty much choose their choice of platform. Have a Linux app but want to sell it to a Windows shop? Virtualization comes to the rescue. Windows applications on a Unix flavor? Again, same deal (though developers might have to pay for a copy of Windows to bundle with virtualized apps if the company doesn't already have a Windows virtual machine running.) But this incentive does not exist for the Macintosh platform. Who would develop a networked server application for the Macintosh platform knowing that you can only sell it to a company that made a big investment in Xserves? Especially since you can just code it for Linux or Windows and let Apple-only shops run it in virtualization.

Additionally, Microsoft has developed an optimized Windows Server 2003 version for virtualization - the Datacenter edition, and various Linux developers have scaled-down versions of the Linux OS for virtualization, including Ubuntu JeOS. We were not able to find a stripped-down version of Apple's Leopard Server. At any rate, running a "full" operating system in virtualization increases overhead and can impact server response times and, therefore, application performance.

Since there's this disincentive for enterprise application developers to develop for the platform, and a comparative performance hit on the platform that should cause network engineers to think twice about the platform, what, exactly is the utility of a MacOSX server?

by Brian Boyko
Editor, Network Performance Daily

Baseline magazine recently put out an article warning IT departments of under-30 "risk takers." Of course. Why not? Everyone knows that the youth are stuck up, and don't fit into corporate culture.

Being 29 years old a week from tomorrow, I was keenly interested (if giddily bemused) in what pejorative things they had to say about us brash young kids who are Ruining-It-For-Everybody™.

"Millenial workers are nearly twice as likely to use personal devices such as cell phones, PDAs and laptops in the workplace as their older counterparts."

Yes, from a security standpoint, an infected laptop or smartphone could provide trouble for the security of the network. But that also means the under-30s are more connected.

Let's look at this from a holistic standpoint. Yes, network security is important. But if personal computers and handhelds provide a more efficient way to get information, they enhance the power of the network. IT is about application delivery - and mobile devices might just be the most efficient way for Young Turks to get to the application. Yes, they can cause problems, but a NOC with automatic reporting can identify the problems quickly enough that the benefits of the always-connected employee outweigh the risks.

"Millenial workers are more than likely to use their work and personal computers for professional and personal use."

I admit that I do this. Sometimes, I need to use my personal Mac to edit video for work, or I need the computer at work to execute a Windows program. But again, this makes me more efficient. I often (after hours or on my lunch break, of course) use my work computer to send personal e-mail, and in fact, is one of the reasons I use Google Mail. Conversely, I log into Exchange to check, and send, work related e-mail from my personal computer at home if something needs my attention.

"Millenial workers believe they should have the right to use software of their choice on their work computers, regardless of its source."

NetQoS has a policy of allowing everyone to install whatever (legal) software they deem necessary to complete the work (as long as it doesn't affect other's network performance). As research for articles I write, I've got a variety of freeware programs, including GIMP, VirtualBox (virtualization software), various video editing programs and, the big one, Firefox, which I downloaded on day one. (I will never understand companies that make you use Internet Explorer in the name of "security")

I've worked at places where we were severely limited in the programs we could have. There's a reason I'm not working at those places anymore - the lack of trust in the ability for a person to choose their own software - their own tools - shows a lack of trust in the ability of the person. And it paints IT as productivity preventer rather than enabler.

This is not to suggest that there should be complete anarchy on the network. But when the IT department locks down everything, it creates more of a productivity hassle than any damage that a virus or hacker can do. There are unsecure apps out there, and the good judgment of the majority does not make up for the poor judgment of the minority. But with that said, why punish the majority for the transgressions of the minority.

It is not, after all, the downloading of malicious apps which affects the network - it is the traffic that those malicious apps produce. Instead of trying to control the application on the desktop (and relying on security on the user/desktop level is futility at best) it's better to control access to the network. You can put computers with out-of-date antivirus or unauthorized apps on an alternative network. You can use anomaly detection to find malicious traffic before it does damage. But there are a whole host of options between application anarchy and resorting to the draconian measures of a culture of complete control.

"While Millenial workers are more likely to visit unauthorized Web sites and install unauthorized applications, they are also more aware of security risks then their Gen X counterparts."

"Millenial workers are slightly more aware of what it takes to secure their apps and devices."

These could easily translate as: "Don't worry. We know what we're doing." And while those have been famous last words a number of times, in this case, I don't think it's ironic. But more importantly, security should never be left to the end-user. We've tried it countless times, it doesn't work. If you rely on defenses at the edge, your network is only as secure as your least security savvy employee.

This next one is very important:

"While all workers want access to technology and devices, each group reports little to no productivity gains as a result. Millenials, however, are more likely to perceive productivity gains from collaboration and Web-based apps." [Emphasis added]

When you think of Web based apps, you immediately consider the network. This is a generation raised on MySpace and Facebook, and Google Maps, and Google Mail, and Google Analytics, and Yahoo Answers, and Wikipedia - this is the generation of the Web based app. These are the tools that the upcoming generation is comfortable with. And the people who develop tools know this. This is why application performance - especially for Web based apps, is so crucial.

In the end, Baseline put out a separate article a few days later, pointing out that under-30s in IT had benefits as well as drawbacks. But why should we believe them? Everyone knows that you can't trust anyone over thirty.