A study commissioned by VoIP provider Inclarity and conducted by YouGov among workers in the United Kingdom found that up to 60% of those polled had, during the previous year, had experienced a full day’s disruption of their company’s phone system. And according to SC Magazine UK, telephone downtime costs UK businesses around £14,000 each day, or, with the current exchange rate, about three billion U.S. dollars per minute.

Worse still, 61% of respondents didn’t have, or weren’t aware of, a disaster recovery or business continuity plan in the event of phone service problems. Considering that kind of money, it is extremely important to have a backup plan for the phones, be able to isolate performance issues, and speed recovery times. Because the only thing worse than one day without phone service is multiple days without phone service. Or baby cheetah murder. I suppose baby cheetah murder is worse.

VoIP is sensitive to latency and jitter, and anything that interferes with either latency or jitter will create bottom-line impacting problems. Now think about all the possible things that can increase latency or cause jitter either within your network or outside of your control. Sobering thoughts, huh?

Monitoring VoIP performance and being warned before problems impact the bottom line is very important, but sometimes, things like hurricanes, tornadoes, backhoes, and little kids pushing the bright shiny red button will happen. it always helps to have a backup plan in place of some sort of catastrophic failure.

Popular Mechanics recently published an article talking about one of the things that frustrates knowledgeable HDTV owners and merely confuses not-so-knowledgeable ones. There is no standard for the amount of acceptable compression on an HDTV signal.

As it turns out, television bandwidth (in the electromagnetic spectrum sense) is very closely related to television bandwidth (in the throughput sense.)

Compression is used to deliver multiple HD channels – the higher the compression, the more channels you can fit either into your slice of the EM spectrum, your satellite bandwidth, or your cable service, you know, whatever multi-billion dollar communications infrastructure you have lying around.

The problem is that in order to deliver more HD channels, some companies apply a level of compression that is destructive to image quality, reducing fine detail and introducing compression artifacts – those magic rectangular blobs you would get from, say, an overcompressed XviD.

In fact, there's no real regulation over high-definition picture quality at all—"none whatsoever," one industry consultant told me. … After all, 1080 lines of poor-quality pixels may technically be "high-definition," but that doesn't mean it looks very good.

One of the most important factors in determining picture quality is bit rate, or how much video and audio data is being sent down the pipe for each program. The technology behind digital television relies heavily on digital compression, and the ATSC specifies that digital TV use the MPEG-2 compression standard, which is also utilized by DVDs, although some satellite broadcasters use the more efficient MPEG-4 advanced video coding (AVC) standard. These compression technologies are necessary in order to deliver a large number of channels to consumers. Without these codecs, an uncompressed HD video stream could require as much as 1 gigabit per second of data capacity—that's 52 times the capacity of the average broadcast channel. With compression, the same stream can be shrunk almost infinitely. But compression is often used overzealously, and picture quality suffers as a result.

In fact, the end picture quality can be so poor that it defeats the very purpose of broadcasting in HD to begin with, as poster “Bfdtv” showed through a series of screenshots on the AVS Forum.

The problem with determining the point at which compression significantly degrades the “quality of television experience” is a highly subjective measure, and differs not only from person to person, but from content to content – high compression is probably fine for talking-head news shows with little movement, but not very good for sports, for example. Crime drama shows might fall somewhere in the middle. Some measure of compression is obviously acceptable, otherwise YouTube, iTunes shows, and yes, pirated movies and TV shows wouldn’t be popular.

I asked Robert Zitter, HBO's chief technical officer, what his company's requirements were. "We do a contractual relationship with all of our distributors, and one of the items that's addressed in there is what they can and cannot do with our signal," he says. "But much of it cannot be quantified. There's not a device that's made that can look at a picture and say pass or fail. We all wish we had it—it would make the negotiations a lot easier. Ultimately, it winds up being more subjective than quantifiable."

I think Zitter may be wrong, however. While this is a difficult problem, it is not necessarily an unsolvable one. We already have a system in place for measuring another “subjective” thing in a “quantifiable way” - voice call quality.

Voice call quality, which is also a “subjective” measurement, is now something that is calculated in VoIP deployments as a Mean Opinion Score [MOS]. Today we can derive the MOS and the subjective end-user experience by tracking all the packet drops, latency, jitter, and general network quality of a VoIP call. MOS for picture quality determined from the amount of macroblocked pixels, the compression rate, and the effective level of detail isn’t far off into the future.

Kim Shorb, Product Manager at NetQoS, and Jeff Hicks, NetQoS VoIP Architect, will be hosting a live Webcast on Tuesday, August 12^th at 10:00 a.m. Pacific time.  They’ll be talking about how to obtain metrics on call signaling and setup protocols, how to gauge the quality of VoIP voice calls, how to show the impact of VoIP on existing applications (and vice versa) by looking at response times, and ensuring adequate bandwidth for VoIP traffic and other business critical applications by determining the volume of VoIP traffic across the WAN. 

As most of you probably already know, VoIP deployments demand optimal network performance to ensure an acceptable quality of experience for the end-user.  I mean, we’ve grown up with telephones our entire lives, and we know how they work, how they sound, and above all, what to expect from them.  Matching the expectations of traditional phone service is daunting because VoIP is sensitive to latency, jitter, and packet loss.  Compound that with the fact that phone service is extremely business critical for any business in the world.

Introducing VoIP apps on your network presents specific challenges. In addition, VoIP can starve out other key business applications and impact productivity. These challenges are best addressed with network-centric performance tools – yet, few VoIP applications offer integration with those tools.   

The transition from analog telephony to VoIP is unlike any upgrade you've ever performed. A VoIP deployment is a major initiative that demands optimal network performance to ensure an acceptable quality of experience for end users. There are many reasons why VoIP is so demanding.  Unlike most data applications, VoIP is sensitive to latency, jitter, and packet loss. However, few VoIP applications offer integrated management tools - the metrics and visibility required to deliver a superior VoIP quality of experience while ensuring network performance. The introduction of VoIP applications on your enterprise network presents specific challenges that can best be addressed with network-centric performance monitoring products.

Join NetQoS experts to learn how VoIP applications have evolved and why they demand optimal network performance. Find out how to take a performance-first approach to the network, and learn why this approach, based on deep insight into end-user quality of experience, will help you ensure optimal call quality.

Additionally, attendees will receive a free copy of the VoIP eBook, "Do You See What I'm Saying? Managing VoIP Quality of Experience on Your Network". This eBook details how VoIP applications work, as well as how to ensure that the applications perform on your network at the highest levels to provide optimal quality of experience to the end-user.

According to Network World, T-Mobile recently announced “T-Mobile @Home” which allows you to buy VoIP-to-the-home service from T-Mobile for $10/mo – assuming you don’t mind the $50 router and the 2 year lock-in contract. 

The interesting part of the article was that T-Mobile released a study that 97% of customers who had a landline service then got a VoIP service - dropped the landline service completely.  The implications are staggering – traffic once reserved for the land lines will now be thrown in with the rest of the Internet mix.  I’m not quite worried about capacity but rather about latency – will latency on the network remain low enough for a VoIP system to remain superior to the traditional phone network even as it grows?

I’m also wondering what to do with all those miles of obsolete land-line copper cable running to all the houses.  Hmm…

By Kevin Davis
Adapted from “Sources of Latency” Whitepaper

When network users call the Help Desk to report poor application performance, you don’t typically hear things like “The router’s CPU is too busy!,” “The network utilization is above 70%!,” or “The carrier path has failed-over to a sub-optimal path.” Instead, what you’re likely to hear is “The network is slow” or “The calls on my IP phone sound terrible.”

Complaints that end-users lodge are nearly always based their quality of experience using the application. And their quality of experience is almost always reliant on time.

Anytime a significant delay occurs in the delivery of network data, application performance suffers. Depending on the type of application and how it works, variances in network delay can have a severe impact on application performance thereby degrading end-user’s experiences.

Two important measurements of time intervals in network transmission systems are referred to as “latency” and “jitter”. Understanding latency and jitter sources and how their values vary in network architectures is critical to engineering application performance and optimizing information resources. For many regular readers, this will be old-hat, but we’ll go over it again.

Network latency is the amount of time it takes for a packet to be transmitted end-to-end across a network and is composed of five variables:

Network Latency = (Distance Delay) + (Serialization Delay) + (Queue Delay) + (Forwarding Delay) + (Protocol Delay)

Serialization Delay refers to the amount of time it takes for a network interface (such as a router’s interface or computer’s NIC) to perform bitwise transmission of a frame unto the outbound media, Forwarding Delay is the amount of time it takes a network device to process a frame/packet by performing a destination address lookup and forwarding the frame/packet to the outbound interface, and Protocol Delay is the amount of time that access or transmission algorithms may contribute to the delay of a network frame, and is typically introduced at the endpoints of the data transmission system.

Serialization delay, on a per-packet basis, becomes insignificant at data rates above 1.544 Mbits/s – or a T1. Forwarding delay is typically insignificant in modern routers and switches (when appropriately configured – significant delay can occur in misconfigured routers.) And Protocol delay typically occurs at the access layer or the end points. So the two major variables that have the most effect on network latency are Distance Delay and Queue Delay.

Distance Delay is simply the minimum amount of time that it takes the electrical signals that represent bits to travel down the physical wire. Optical cable sends bits at about ~5.5 µs/km, copper cable sends it at ~5.606 µs/km, and satellite sends bits at ~3.3 µs/km. (There are a few additional microseconds of delay from amplifying repeaters in optical cable, but compared to distance, the delay is negligible.)

Distance delay can have a significant impact on application performance for applications that require a large number of network round trips in order to complete a transaction – for example, custom transactional based applications, database queries, and VoIP, which begins do degrade when one-way end-to-end latency exceeds 200-220 milliseconds.

One of the biggest sources of end-user ire are database queries designed to run over a LAN ported to the WAN. For example if a user executes a SQL database query that requests 100 rows of a database table, one row at a time, over a link with a latency due to distance of 60 ms, it would take approximately 6 seconds (60 ms * 100 turns) to complete the transaction. The same query executed by a user on a LAN connected to the same database server would take less than 2-3 ms to be completed, as the latency due to distance across the LAN is insignificant.

Queue Delay is the amount of time a packet must spend in a network buffer waiting its turn to be transmitted. Network interfaces transmit one frame at a time, typically one bit at a time. As such, when two or more packets are forwarded to a network interface at the same time, or close to the same time – one packet is transmitted while the others are put in a queue on the interface buffer to await their turn at the interface. Packets that are put into the queue must wait until they can be transmitted, adding milliseconds of delay.

Increases in Queue Delay can be measured and detected by monitoring traffic along a given network path. Typically, most intermittent increases in latency above the baseline distance latency can be attributed to network congestion. (In order to reduce the possibility of excessive queue delay, application servers that are members of the same application architecture should be placed on the same Ethernet switch and on the same VLAN to ensure they do not have to compete for uplink bandwidth when problems like the one pictured above occur.)

Worse still, if the problem gets worse and packets wait in increasingly longer lines within the queue, the buffer may become full and the packets may be dropped. Packet drop, in turn, causes TCP connections to throttle back on the rate of transmission.

Those are some of the main causes of latency – but what about jitter?

Jitter is a term that refers to the variance in the arrival rate of packets from the same data flow, and abnormal jitter values can negatively impact real-time applications like VoIP and video. Jitter is typically created by three different mechanisms in a network: variance in Serialization Delays due to variance in packet sizes, variance in per-packet Queue Delay due to packet spacing from multiple sources at a common outbound interface, or packets taking different routes from source to destination – perhaps due to per-packet load sharing or routing issues.

The most effective way to deal with jitter is by using low-latency queuing for VoIP and video traffic on network interfaces with large serialization and/or queue delays. In addition, endpoints (such as IP phones) can use jitter buffers or playout delay buffers in order to deliver received packets at a constant rate to the end consumer. These buffers are typically 30-50 ms in depth, and thus they attempt to manage jitter values within these values on any single one-way path. While these buffers technically add 30-50ms in latency, they significantly reduce jitter. Since human beings don’t start to notice latency in VoIP or VideoIP applications till it hits about 200ms, if latency can be kept to under 150 milliseconds, then jitter can be significantly reduced using this method.

by Brian Boyko
Editor, Network Performance Daily

Ars Technica reports that the National Institutes of Health released a study which show that wireless-phone only households are increasing – currently 15.8 percent of households. 

But that’s just part of it – other consumers are switching to VoIP services – gobbling up another 13.8 percent of U.S. households. 

There are certainly economic factors which result in this change - me, I cancelled my landline phone service when I realized I was paying less on a month-to-month basis on my cellphone than I was for a service I barely used.  Since then, I’ve moved around a lot (seven times in the past five years) and keeping the same cellphone instead of canceling and re-enabling service makes a hell of a lot of sense. 

But still – three out of ten consumers?  This is a major shift – and one that’s likely to continue as cellphone and VoIP quality gets better. 

Of course, there will be more demand for VoIP networking in the future – and with it, a need to monitor VoIP quality of experience.  But more than that is the idea that there is an entire infrastructure of phone lines – stretching from sea to shining sea and beyond – that connect the last mile to the local phone exchange. 

It seems like such a waste. 

But wait!  DSL service only uses the 25kHz and above part of the spectrum – 4kHz is reserved for voice.   If landlines are repurposed ONLY for data, with VoIP being another application on the all-data network, that could free up the 4kHz spectrum currently being used.  Maybe we can use that 0-4kHz band for broadband to rural homes – which can clearly get 4kHz data if they can get a phone call. 

This is especially important for rural broadband penetration.  The longer the distance to the exchange, the lower the quality of high bandwidth exchanges – this is why your DSL service gets slower the further you are from your local phone exchange.  But the 4kHz currently used for voice can travel much greater distances – it won’t be as fast as the DSL available in the cities but repurposing the 4kHz bandwidth from voice to data might make a huge difference to getting some minimal broadband to the most rural parts of the world.

Now, this won’t make a whole lot of difference to a person living in the city – DSL works by dividing that 25kHz-and-up portion of the spectrum into 4kHz chunks, each one connecting with a speed equivalent of a modem.  It is the multitude of these channels – hundreds in most cases – that makes DSL speed possible.  Repurposing the broadband of 0-25kHz would result in only six additional channels.  Assigning two for upload and four for download, you’d have speeds of around 14.4kBytes/s (or 115.9kbits/s) upload and 28.8kBytes/s (231.3kbits/s) download.  That’s not much of a speed boost. 

Still, if you’ve been plodding along on a “56.6k” modem, at speeds of 7.2kBytes/s, this would be like an oasis in the desert.  And what about those phone calls?  Well, if you make the same phone calls with VoIP that you were with the standard 0-4kHz landline, it would only take about 20.8kbits/s using the G.723.1 codec – that still leaves you with 80% of your broadband capacity when on the phone – and 100% of your broadband when you’re off it.  For someone whose only current Internet connectivity choice is a modem, currently getting 16% of a theoretical data capacity – and 0% when you’re on the phone – that’s a major improvement.   

What do you think?  It seems reasonable, but there might be a flaw in my math – I did only pass Calc I on my third try.  Let me know if you have something to share in our comments section.

We're releasing NetQoS VoIP Monitor v1.1. Biggest changes: SIP (Session Initiation Protocol) support, automatic and on-demand problem investigation, and capacity planning reports.

I want to start with SIP support, because there's an interesting related story that caught my attention when it came out on Slashdot.

One of the odd things about SIP is that it is, to some extent, a peer-to peer based protocol. The advantage of this is that it only requires a simple core network, with all the fiddly bits distributed to the network edge. This makes SIP more scalable than other protocols. You can see why our customers think SIP support is important and why NetQoS worked to put it into this release.

But as a side effect of the way the technology was designed, SIP's peer to peer network means that it can be difficult to route emergency calls because of the mobility of IP endpoints and the fact that SIP has no network location capability - you'll remember that Vonage got into a little bit of trouble a while back because it couldn't consistently promise E911 support. (That has since been fixed.)

SIP also establishes a VoIP connection directly between the two calls out at the edge. Once the call is set up, the data does not pass through any sort of central server owned or controlled by the VoIP provider. That makes it harder for the government to legally (or whatever) intercept calls.

I mention this because the actual documents governing the rules behind U.S. government interception of VoIP was leaked to Wikileaks on the 15th of March. Now, this is nothing new - CALEA requires VoIP providers to maintain wiretapping capability - just like the plain-old-telephone-service providers are. It's interesting, however, to see the documents. Or at least it might be to somebody else who is interested in network security and encryption.

But from a performance angle, the CALEA requirements for wiretapping are directly in opposition to the efficiency of a SIP VoIP network - that is, if a service provider must be in the middle of every call, it eliminates the benefits of the P2P structure. That adds a lot of network overhead.

The other new features in VoIP Monitor v1.1 are generally less conversation sparking - but no less important. Most of our other products, such as SuperAgent, have both automatic and on-demand problem investigation and capacity planning reports. These capabilities have been added to VoIP monitor in the new version.

Automatic investigations occur when a VoIP performance threshold - such as delay to dial tone - is exceeded. Then VoIP Monitor traces the call signaling path and compares it to the automatically generated baselines.

As far as capacity planning reports helps go, VoIP Monitor v1.1 providers enhanced reports on call volume, call quality, call failures, grade of service and gateway utilization. It provides a view of the effect of different call volumes. With this information, IT professionals can view capacity for specific locations or gateways or for the enterprise as a whole - the utilization reports are especially useful when negotiating contracts with service providers.

We have a demo of VoIP Monitor v1.1 up and running at VoiceCon in Orlando, at booth #1305, if you're attending.

You can’t measure the quality of a VoIP call without having a set of metrics. While traditional TCP application performance metrics – like transaction time and throughput – are important, they may be difficult to relate to the user experience with the VoIP phone system. There are some user experience metrics that relate directly with call setup performance, however.

First, there’s the delay to the dial tone. In a VoIP system, the phone has to send a message to the call server letting the server know that the phone is off the hook. The call server then sends a message back to the phone telling it to play the dial tone. The time it takes for the message to be sent and the response received is the delay to dial tone.

Continue reading "VoIP Management Series: Key VoIP Call Setup Metrics" »

It’s important to understand each call setup protocol in a VoIP system, because each different call type can involve different components and different protocols.

First, there is On-Net calling, which takes place between two IP phones on the same logical network. In this scenario, the phones use setup protocols like SIP or SCCP to interact with a call server to set-up and take down each phone call. These types of calls are entirely IP based – they’re carried on the network and do not have to go out to the PTSN.

The simplest On-Net scenario, Intracluster Calling, means that both phones are on the same call server or cluster of call servers.

Continue reading "VoIP Management Series: VoIP Call Setup Protocols" »

Often when looking at VoIP quality, people focus on what happens after your call goes through – crackles, drop-outs, and delay aren’t fun, but they’re only half of what you need to worry about with VoIP rollouts.

The VoIP experience begins not with “Hello” but with “eeeeeeeeeeeee.” It begins when you pick up the handset and get a dial tone. The quality of experience of the overall phone systems is shaped not only by the quality of the voice on the other end, but also the availability that the system provides.

Continue reading "VoIP Management Series: VoIP Call Setup Performance" »