This blog post is about the integration of metrics from multiple F5 devices into the NetQoS Performance Center. When researching this article, I found out that this means that NetQoS Performance Center can import performance metrics from F5 Networks BIG-IP Local Traffic Manager and BIG-IP Global Traffic Manager, which provide services for managing application security, application acceleration, and network availability. However, I was disappointed to learn that this integration has nothing to do with F5, the heavy metal band that features former Megadeth bassist Davide Ellefson and drummer Jimmy DeGrasso.

Instead of logging into individual F5 devices for performance information, IT personnel can get visibility into f5 metrics through a single console without needing f5 administrative capabilities or a bowl of M&Ms with the brown ones removed.

By integrating with F5, NetQoS can help simplify the increasingly complex challenge of monitoring network and application performance across an organization. For joint customers this means a single view into an unlimited number of F5 devices and detailed performance metrics delivered in real time through the NetQoS Performance Center.

So while this integration will help more quickly address network problems and improve application delivery – with the Performance Center’s analysis and alerting capabilities combined with the F5 reporting – it will not, by any stretch of the imagination, produce a technically expert melodic bass solo at any time.

The combination enables IT professionals to better plan investments in new content delivery hardware, accurately size and configure new IT deployments, verify the impact investments have on application delivery, and solve performance issues in complex data center environments, but will not be included on a compilation CD with Rob Zombie, Queensryche, Metal Church, et. al.

NetQoS demonstrated the integration at the F5 Partner Summit (not to be confused with Ozzfest) yesterday in New Orleans.

Popular Mechanics recently published an article talking about one of the things that frustrates knowledgeable HDTV owners and merely confuses not-so-knowledgeable ones. There is no standard for the amount of acceptable compression on an HDTV signal.

As it turns out, television bandwidth (in the electromagnetic spectrum sense) is very closely related to television bandwidth (in the throughput sense.)

Compression is used to deliver multiple HD channels – the higher the compression, the more channels you can fit either into your slice of the EM spectrum, your satellite bandwidth, or your cable service, you know, whatever multi-billion dollar communications infrastructure you have lying around.

The problem is that in order to deliver more HD channels, some companies apply a level of compression that is destructive to image quality, reducing fine detail and introducing compression artifacts – those magic rectangular blobs you would get from, say, an overcompressed XviD.

In fact, there's no real regulation over high-definition picture quality at all—"none whatsoever," one industry consultant told me. … After all, 1080 lines of poor-quality pixels may technically be "high-definition," but that doesn't mean it looks very good.

One of the most important factors in determining picture quality is bit rate, or how much video and audio data is being sent down the pipe for each program. The technology behind digital television relies heavily on digital compression, and the ATSC specifies that digital TV use the MPEG-2 compression standard, which is also utilized by DVDs, although some satellite broadcasters use the more efficient MPEG-4 advanced video coding (AVC) standard. These compression technologies are necessary in order to deliver a large number of channels to consumers. Without these codecs, an uncompressed HD video stream could require as much as 1 gigabit per second of data capacity—that's 52 times the capacity of the average broadcast channel. With compression, the same stream can be shrunk almost infinitely. But compression is often used overzealously, and picture quality suffers as a result.

In fact, the end picture quality can be so poor that it defeats the very purpose of broadcasting in HD to begin with, as poster “Bfdtv” showed through a series of screenshots on the AVS Forum.

The problem with determining the point at which compression significantly degrades the “quality of television experience” is a highly subjective measure, and differs not only from person to person, but from content to content – high compression is probably fine for talking-head news shows with little movement, but not very good for sports, for example. Crime drama shows might fall somewhere in the middle. Some measure of compression is obviously acceptable, otherwise YouTube, iTunes shows, and yes, pirated movies and TV shows wouldn’t be popular.

I asked Robert Zitter, HBO's chief technical officer, what his company's requirements were. "We do a contractual relationship with all of our distributors, and one of the items that's addressed in there is what they can and cannot do with our signal," he says. "But much of it cannot be quantified. There's not a device that's made that can look at a picture and say pass or fail. We all wish we had it—it would make the negotiations a lot easier. Ultimately, it winds up being more subjective than quantifiable."

I think Zitter may be wrong, however. While this is a difficult problem, it is not necessarily an unsolvable one. We already have a system in place for measuring another “subjective” thing in a “quantifiable way” - voice call quality.

Voice call quality, which is also a “subjective” measurement, is now something that is calculated in VoIP deployments as a Mean Opinion Score [MOS]. Today we can derive the MOS and the subjective end-user experience by tracking all the packet drops, latency, jitter, and general network quality of a VoIP call. MOS for picture quality determined from the amount of macroblocked pixels, the compression rate, and the effective level of detail isn’t far off into the future.

You could say that San Franicsco had a network security issue, but that’s really not accurate. San Francisco’s network was secure. It’s just that it was secured against the wrong people.

Though the San Francisco story of Terry Childs is nearing a conclusion, it’s important to look back at this and see what lessons we can learn from it. Quite frankly, being able to learn something from it seems the only upside to this debacle.

If you’re not familiar with the story, Terry Childs was a network administrator for the City of San Francisco. He built the network from the ground up, maintained it, and was the sole rights manager for the network. But Childs supposedly felt so protective of his network that he allegedly decided not to give out those root passwords, configuration changes, and documentation after he was dismissed to colleagues that his lawyer claims: “had in the past maliciously damaged the system themselves, hindered his ability to maintain it, and shown complete indifference to maintaining it themselves.”

In other words, the network was Childs’ baby, and he wasn’t about to give it up to people he thought incompetent.

According to ComputerWorld, this situation is not rare in IT:

"Unfortunately, it is not that uncommon to come into a situation where one or two people have created a situation where not only are they the only ones that know what is going on, but they are the only ones that can do anything," said Lou Michael, director of network and infrastructure services in Virginia's Arlington County department of technology services.

The issue isn't just control over passwords, but also over documentation relating to configurations and changes. Often in situations such as this, "requests for access, passwords and documentation are frequently taken as hostile acts by those that have been holding the keys to the kingdom," he added. "In my experience, I have encountered this type of situation on more than one occasion," he said. In one incident, a mainframe systems programmer had to be fired for changing access rights because he disapproved of others' activities on the system, Michael said. In another case, the individual resigned when he "realized that the pressure to follow processes and procedures was not going to go away despite the protesting," Michael said.

In IT we do have a tendency to form attachments to the tools of our trade. It can be difficult, as network engineers know themselves to be artists and the maintainers of a complex system, one in which balance and perfection are key.

The grand irony is that much of what IT does is invisible to most people in the company, and the rest of what IT does is incomprehensible to most people in the company. Even if they see the grand opus, they don’t recognize it for the masterpiece that it is. Pigeons crapping on sculpture.

We’ve mentioned before the mistrust that occurs between network and application engineers and management, and how it can be a hard sell to give management even high level visibility into network performance. But network engineers need to also realize their role of supporting the business and help break down the silos between them and everyone else.

Otherwise, you end up with situations like the one that happened in San Francisco.

Kim Shorb, Product Manager at NetQoS, and Jeff Hicks, NetQoS VoIP Architect, will be hosting a live Webcast on Tuesday, August 12^th at 10:00 a.m. Pacific time.  They’ll be talking about how to obtain metrics on call signaling and setup protocols, how to gauge the quality of VoIP voice calls, how to show the impact of VoIP on existing applications (and vice versa) by looking at response times, and ensuring adequate bandwidth for VoIP traffic and other business critical applications by determining the volume of VoIP traffic across the WAN. 

As most of you probably already know, VoIP deployments demand optimal network performance to ensure an acceptable quality of experience for the end-user.  I mean, we’ve grown up with telephones our entire lives, and we know how they work, how they sound, and above all, what to expect from them.  Matching the expectations of traditional phone service is daunting because VoIP is sensitive to latency, jitter, and packet loss.  Compound that with the fact that phone service is extremely business critical for any business in the world.

Introducing VoIP apps on your network presents specific challenges. In addition, VoIP can starve out other key business applications and impact productivity. These challenges are best addressed with network-centric performance tools – yet, few VoIP applications offer integration with those tools.   

The transition from analog telephony to VoIP is unlike any upgrade you've ever performed. A VoIP deployment is a major initiative that demands optimal network performance to ensure an acceptable quality of experience for end users. There are many reasons why VoIP is so demanding.  Unlike most data applications, VoIP is sensitive to latency, jitter, and packet loss. However, few VoIP applications offer integrated management tools - the metrics and visibility required to deliver a superior VoIP quality of experience while ensuring network performance. The introduction of VoIP applications on your enterprise network presents specific challenges that can best be addressed with network-centric performance monitoring products.

Join NetQoS experts to learn how VoIP applications have evolved and why they demand optimal network performance. Find out how to take a performance-first approach to the network, and learn why this approach, based on deep insight into end-user quality of experience, will help you ensure optimal call quality.

Additionally, attendees will receive a free copy of the VoIP eBook, "Do You See What I'm Saying? Managing VoIP Quality of Experience on Your Network". This eBook details how VoIP applications work, as well as how to ensure that the applications perform on your network at the highest levels to provide optimal quality of experience to the end-user.

Recently, we did an unscientific (and really, I cannot stress that word enough) but real-world test of performance using the Ono plugin for BitTorrent client Vuze/Azureus. Our results were inconclusive.

David Choffnes, the author of the Ono plugin, responded to the test in our comments section of that article:

Regarding your results, it is difficult to run controlled experiments because even when you download the same torrent, you're doing it at different times with necessarily different swarms. My research group's evaluation is not limited by this, and we showed that performance improves *on average*.

Also note that if Ono doesn't find any nearby peers, it can't help your performance. You can see if Ono found nearby peers (and is using them) in the "Ono" plugin view … Also, the plugin's effectiveness is limited by the fact that "only" 180,000 users have installed Ono. The more people use it, the more likely you'll find nearby peers.

One last point -- even when Ono doesn't dramatically improve performance, it encourages better "Internet citizen" behavior. Why transfer data from halfway across the world when you can get the same data and (potentially better) performance from peers nearby? Ono makes it easier to do the latter, which should eventually help everyone using the Internet.

Ono is part of Aqualab, a Northwestern University computer science project researching large-scale distributed computing. Choffnes, a doctoral candidate, will present his findings at SIGCOMM next month, and his paper on the subject can be found here – which is great if you like trigonometric functions in your technical literature.

There’s also a telling paragraph which may explain why we got the results we did for our tests (other than just the random variability of different BitTorrent swarms), instead of a massive throughput boost.

In our analysis, we compare statistics from peers located by Ono (referred to as Ono-recommended peers) to those from all peers selected at random by the BitTorrent protocol, which also includes those located by Ono.

In Network Performance Daily's analysis, we compared statistics from peers located by Ono combined with peers selected at random from the BitTorrent protocol, against only peers selected at random from the BitTorrent protocol.

To determine the cosine similarity value for a peer, Ono must be able to compare its ratio maps with those of other peers. The latter information can be obtained in a number of ways: through direct exchange between peers, from distributed storage and from trackers. Ono currently supports the first two options. With direct exchange, when two peers running the Ono plugin perform their connection handshake, the peers swap ratio maps directly… Though Ono enjoys a large user base, it is still a small fraction of the total BitTorrent population. Thus Ono also attempts to perform DNS lookups on behalf of other peers that it encounters, to determine their ratio maps. This enables Ono to perform biased peer selection over a much larger set of peers, including those not running the Azureus client. From both direct exchange of ratio maps and DNS lookups, our Ono clients locate over 180, 000 peers per day using our CDN-based approach.

When Ono determines that a peer has similar redirection behavior, it attempts to bias traffic toward that peer by ensuring there is always a connection to it, which minimizes the time that the peer is choked. Due to limitations of the Azureus plugin API, we are currently unable to bias other aspects of peer connections, e.g., the bandwidth allocated to each connection.

In addition to Ono, Aqualab also does other projects that are designed for improving Internet performance in a number of other areas. Choffnes’s advisor, Dr. Fabian Bustamante, has been working on "sustainable scalability in distributed systems,” called the 3R project. Many P2P and internet VoIP systems are built in a way that routing is controlled at the application layer, and that in order to identify better paths and faster throughput, the application probes the network environment repeatedly, as the application has no quick way to determine whether a particular peer or node is performing well except by trying to connect to it. The 3R project seeks to decrease probing by re-using the view of the network gathered by long-running, ubiquitous services.

While enterprise networking and Internet networking are two different beasts, performance advances in one usually lead to advances in the other, and with cloud computing promising to make enterprise networking a hybrid of LAN, WAN, and Internet connectivity, these advances remain important.

According to this recent article in ComputerWorld:

IT staff jobs are at increasing risk -- both for contractors and in-house workers -- according to a survey of top CIOs by Goldman Sachs & Co. released last week. Global services companies will also feel the pinch because of the slowing economy.

My reaction:

According to the article, the study showed that CIOs are:

• Looking to cut resources from contracted IT staff


• Not looking to start or fund “discretionary IT projects”


• Looking for solutions with a “high and fast ROI.”


• See the “greatest potential for cost reduction in IT in the area of networking equipment”


• See server virtualization and server consolidation as their top two priorities, with data center consolidation an additional priority.


• Do not see cloud computing as a priority

Let’s talk about a couple of these. First, the “high and fast ROI.”

It is notoriously difficult to prove IT’s return on investment. There is some truth to the idea that IT is a utility to big companies despite the fact that they do see it as a necessity. They don’t consider electrical power a profit generator either, but without it, the business grinds to a halt.

And unless you work in a software company, chances are that IT isn’t a “profit-generating center.” Troubling.

However, what IT can do is lower costs for the company as a whole. (Indeed, the theory of having all those newfangled computers in the first place is that it saves a fortune on cross-country pneumatic tubing and hundreds of thousands of file clerks sitting on typewriters. Not to mention all the wite-out you’d need). The trick is having a way to prove that you’re lowering costs and to quantify exactly which costs you’re lowering. For that, you need a way to baseline performance and a way to determine what the effect of each IT project and change was.

Additionally, being able to detect and track detrimental changes to the network before people start calling into the help desk shows in a more anecdotal sense the utility and power of a well-functioning IT department.

This is especially important when you consider server consolidation – powered by virtualization, running many servers on one box decreases costs but certainly increases complexity. Being able to rapidly detect problems as they occur not only can decrease mean time to repair, but helps isolate problems. In a complex system, little problems early in the chain tend to cause big problems later on down the chain.

With server consolidation comes data center consolidation – seen as a major cost-cutting measure. But what that does is increase the amount of traffic that’s traveling along low-bandwidth WAN links instead of high-bandwidth LAN links. All things are relative – as you bring the servers closer to home, you’re also moving the users farther out. Being able to monitor end-to-end performance in these conditions – and figure out ways to improve performance on the WAN, is critical.

The one big silver lining in this is that cloud computing is not yet a priority – meaning that there’s still room for in-house IT even as belts get tightened. Focus will be, it seems, on the LAN, WAN, and DAN, er, if you are a network engineer named Dan, for example, our Dan in our IT department. (I suppose that you could worry about LAN, WAN, and BILL, or LAN, WAN, and LAURA but neither of those rhyme.)

Guest Post
by Josh Hinkle
Manager, Network Management & Security,
American Heart Association

As the youngest of three siblings I recall my brother hating to give my sister a ride to and from school. Even worse, he despised having her butt-in when he was hanging out with his friends.  After all, he was cool and his little sister was well…his little sister.

For my parents this was a great solution because they didn’t have to be the full-time taxi service anymore. Older siblings despise this role as chauffer because their younger siblings end up riding the coattails of older siblings to after school social activities.

At first glance I felt like Web 2.0 was that younger sibling tagging along on the years of hard work by global IT – built on an existing infrastructure while showing the ability to become popular seemingly overnight.

I spent the last 12 years in Information Technology with an emphasis in network management, eight of those years at the American Heart Association.  Most recently, I’ve served as Manager of Network Management & Security at the AHA the last two years. Like most corporate network managers I have a vested interest in enterprise application delivery. Our business, like many others, depends on enterprise applications being access by thousands of staff in hundreds of locations. At times our staff has been challenged with latency and remote connectivity. It was then we turned to NetQoS to measure, alert, report and trend our network traffic in an effort to take operations to the next level. As those processes recently began to mature our attention shifted to the free-riding sibling Web 2.0.

While Web 1.0 paved the way for networking billions of people, Web 2.0 is stealing the thunder. In a matter of months everyone has seemed to get LinkedIn, gotten poked on Facebook or Twittered someone. Web 2.0 is now carpooling with enterprise traffic across the same infrastructure competing for the same popularity of bandwidth.

AHA revolves around providing information to reduce cardiovascular disease and stroke, and Web 2.0 has increased the demand on AHA’s infrastructure.  It provides a low investment to a large audience.  Certainly, Web 2.0 has the potential inform and collaborate with millions, but the background costs of infrastructure and man hours concern me.

Web 2.0 apps are not representative of the traditional enterprise applications.  First, they exist outside the bounds of the enterprise infrastructure, yet we manage them on the same WAN. Second, the interactive nature of Web 2.0 apps require additional bandwidth.  And third, Web 2.0 applications are not unlike a “human machine” that grows with every click.

Right now, the American Heart Association is engaging  in a Social Media Evaluation project to determine where and how we can further leverage this new platform;  currently we are leveraging an application in Facebook to reach new audiences interested in the American Heart Association's Start! Walking Movement. The American Heart Association’s “You’re the Cure” Network has a Facebook site coordinates volunteer efforts to inform public officials.  Our TCS (Technology and Customers Strategy) Department started an AHA Technology Blog to discuss the technology we use and the organizational accomplishments achieved using technology.  Most recently, we posted a story about how a customer Googled symptoms he was having, which led him to our site on heart attacks.  His doctor told us that he called 911 immediately and survived because of it.

What I’m currently proposing to senior management is for AHA is to manage our network as if it were two separate networks – one network for our two very different needs.  The first network would use MPLS and provide managed bandwidth prioritizing queues for enterprise applications, and the second would offload all Internet bound traffic from the first. 

Not too long ago this type of infrastructure investment would appear to be unjustifiable, but given new trends in Web 2.0 as a platform and evolving cost structures it may very well be a business driving reality. We need a network as flexible and adaptive as the business demands.

This will increase our costs for transport but we are now able to guarantee Enterprise traffic on one network and adapt to evolving trends like Web 2.0, video conferencing, etc. on the other.  Even with the added costs, by negotiating more volume into our transport cost contracts, we can lower our per MB costs. 

Not all of the changes are measured in the bottom line however.  Our applications should see great gains in performance, and our network will be fully redundant for each site as the MPLS will failover for Internet traffic, and vice-versa. 

I must admit, at first I considered Web 2.0 an (admittedly exciting) nuisance in my network and a menace to my plan for enterprise application delivery.  But recently I created my own blog, linked my social network sites, posted you tube videos and started speaking a second language of Web 2.0 terminology.  I matured in my thinking as a network manager and now I embraced the qualities of web 2.0 much like my siblings and I matured in our appreciation for each other.

Web 2.0 may be the sibling that is bumming a ride but it has its qualities to appreciate; it may even mature into a traditional enterprise operations model.  Fasten your seatbelt and make the most of the ride.

In this short video, NetQoS CEO Joel Trammell talks about gas mileage improvements and latency improvements and why the graph tends to be a hyperbolic curve. 

Warning: Information contained herein is not actual medical advice and should not be used to aid a fallen co-worker suffering from a heart attack. However, in certain cases, it may prevent high blood pressure, headache, anxiety, depression, anger, and insomnia.

An old saying in IT: You can’t manage what you don’t measure. Or as one of our customers told me, “one test is worth 1,000 opinions.” If you subscribe to those ideas, and you’re a customer of ours, I have good news. NetQoS customers have a wide array of data at their fingertips to conduct a meaningful “CPR” session.

• Check Status – Executive or management level visibility into performance of application response times, device level statistics, or network traffic composition.


• Plan Ahead and Avoid Problems – Baseline current performance, make and justify investment decisions to improve performance. Our customers are using our tools to gauge


• React and Recover – Identify hot spots and perform rapid trouble isolation and resolution.

(By the way, this is not to be confused with our services CPR engagement, which is "Critical Problem Resolution." Aren’t mnemonics neat?)

Of course, there aren’t any network or application engineers who don’t want tools to help them plan new work activities and respond effectively when faced with a difficult problems. But there’s often a bit of hesitation when it comes to management reports – a prevailing attitude is that giving management level visibility into the network is a bit like handing someone a stick and asking them to beat you roughly about the head and shoulders.

Certainly, that’s one way of looking at it. But, I’ve always found that somehow and in some way, performance management information is going to “get out”. Someone will complain that the network sure is slow. Or your boss may get blind-sided in a dark hallway by an angry user.

Or, alternatively, you can provide performance visibility to management and enlist their support in fixing any problems you find.

As network and application engineers, it’s easy to think of yourself as doctor to the network. No doctor, however, would conceal high-risk factors for heart disease from a patient.

A note from Joel Trammell, CEO, NetQoS.

Recently, there was an article by technical marketing manager Ben Erwin about why the company is named NetQoS, talking about the importance of Quality of Service policies.  While the message about quality of service as it is known today is important, there are a few things that should be corrected.

See, “Quality of service,” in the field of telephony, was defined in 1994 in the ITU-T Recommendation E.800 as the collective effect of service performance which determines the degree of satisfaction of a user of the service.

At the time of this definition it was believed that ATM (Asynchronous Transfer Mode) would be the future networking technology that would allow for combining voice, video and data traffic on the same circuit. ATM had by design many sophisticated traffic engineering approaches available within the protocol for ensuring QoS.

Over time IP (Internet Protocol) became the key competitor to ATM as a standard network protocol. IP’s only traffic engineering capability was the ability to provide differentiated classes of service. In an attempt to make IP seem similar to ATM, IP proponents began using the term QoS instead of the more technically correct term CoS (Classes of Service) to describe IP’s capabilities.

As IP won out in the marketplace over ATM a whole generation of engineers have come to believe that QoS and CoS are equivalent terms. NetQoS was named with the original, broader definition in mind.

(Also, the domain name NetQoS.com was available.  That was a big part of it too.)