Series Summary and Editorial
Part One: Interview with Texas State Rep. Joe Driver
Part Two: Interview with Matt Miller, Institute for Justice
Part Three: Interview with Capt. RenEarl Bowie, Texas Private Security Bureau

by Brian Boyko
Editor, Network Performance Daily

We’ve written three stories and conducted three interviews regarding HB2833.  The first was with the author of the law, Texas State Representative Joe Driver, the second with Matt Miller of the Institute for Justice, and the third with Texas Private Security Bureau Captain RenEarl Bowie.

Here is our editorial summary:

HB2833, the law designed to make changes to laws regarding private investigation but has PC and Network techs worried that their work may now be illegal, has caused confusion and worry from normal people doing normal jobs in a normal manner.  Whatever the original intent of the law, it is clear to see from its effects that the law itself is poorly written. 

Ultimately, words like “open to interpretation” and “case-by-case basis” are not words you want to use when describing either the meaning of, or enforcement of, the law.

So, where did things go wrong?  I think the man problem was a key misunderstood concept by Texas State Rep. Driver when he wrote the law.  It is clear from the interview with him that he believes that there is a clear and well defined line between “retrieval of data” and “investigation.”

“’Review, analyze, and investigate’ are the three key words, in my opinion, that drive the need for people to have some kind of license. Because if they're doing some of that, then they don't need to be - it doesn't need to be just anybody able to do that - they need to have somebody that has a security license. But if someone's just retrieving information and providing information for someone who is going to analyze, to use one of the words, then that's just a regular computer repair person.” – Rep. Driver.

But what Rep. Driver simply did not realize is that in the practical realities of IT, no such line exists. Any and every interaction that any IT person has with a computer requires some sort of “review, investigation and analysis,” whether it’s simple troubleshooting or complex network latency optimization. 

I can see where Rep. Driver was going with the law and what his intent was when writing it – rooting around through someone’s Windows Recycle Bin can be just as invasive as rooting around in somebody’s trash. 

But rooting around in the guts of a computer to discover the cause of a malware infection is different from rooting around in the guts of a computer to discover infidelity.  However, instead of making the criteria of “investigation” the purpose and use to which the information could be put, the law makes the criteria the way that the information is stored – “computer-based data not available to the public.”  The end result is that the net was cast too widely. 

Compounding this problem is the interpretation provided by the Texas State Private Security Bureau of the law – a literal one.

“Computer repair or support services should be aware that if they offer to perform investigative services… they must be licensed as investigators” – Texas Private Security Bureau Opinion Summary.

Unlike the law itself, the opinion summary is an unambiguous statement, and while Capt. Bowie may say that the law will be interpreted on a “case-by-case” basis, that is not what is in the official statement of opinion. 

As for the court case brought by the Institute for Justice – unfortunately, the Institute for Justice seems to want to fight this case on Constitutional grounds.  However, that will be a hard sell, as qualifications and licensing are clearly powers that states exercise, from state bar associations for lawyers, and state medical boards for doctors.  If the state of Texas wants to make a PI license a requirement for PC repair techs, it certainly has the power to do so.  It may be absurd, but absurdity is not unconstitutional. 

So, where does that leave technical practitioners like network engineers and PC repair gurus?  As a practical matter, I think most people are going to continue going about this, “business-as-usual” style and make a stink only after the law is enforced on some, most likely unsuspecting, tech somewhere in Texas.

The good news is that I think that it is indeed possible to clarify and change the law through the legislative process – Rep. Driver has stated that he would indeed make changes to the law if it needs clarification or amendment. 

It clearly does.  

By Kevin Davis
Adapted from “Sources of Latency” Whitepaper

When network users call the Help Desk to report poor application performance, you don’t typically hear things like “The router’s CPU is too busy!,” “The network utilization is above 70%!,” or “The carrier path has failed-over to a sub-optimal path.” Instead, what you’re likely to hear is “The network is slow” or “The calls on my IP phone sound terrible.”

Complaints that end-users lodge are nearly always based their quality of experience using the application. And their quality of experience is almost always reliant on time.

Anytime a significant delay occurs in the delivery of network data, application performance suffers. Depending on the type of application and how it works, variances in network delay can have a severe impact on application performance thereby degrading end-user’s experiences.

Two important measurements of time intervals in network transmission systems are referred to as “latency” and “jitter”. Understanding latency and jitter sources and how their values vary in network architectures is critical to engineering application performance and optimizing information resources. For many regular readers, this will be old-hat, but we’ll go over it again.

Network latency is the amount of time it takes for a packet to be transmitted end-to-end across a network and is composed of five variables:

Network Latency = (Distance Delay) + (Serialization Delay) + (Queue Delay) + (Forwarding Delay) + (Protocol Delay)

Serialization Delay refers to the amount of time it takes for a network interface (such as a router’s interface or computer’s NIC) to perform bitwise transmission of a frame unto the outbound media, Forwarding Delay is the amount of time it takes a network device to process a frame/packet by performing a destination address lookup and forwarding the frame/packet to the outbound interface, and Protocol Delay is the amount of time that access or transmission algorithms may contribute to the delay of a network frame, and is typically introduced at the endpoints of the data transmission system.

Serialization delay, on a per-packet basis, becomes insignificant at data rates above 1.544 Mbits/s – or a T1. Forwarding delay is typically insignificant in modern routers and switches (when appropriately configured – significant delay can occur in misconfigured routers.) And Protocol delay typically occurs at the access layer or the end points. So the two major variables that have the most effect on network latency are Distance Delay and Queue Delay.

Distance Delay is simply the minimum amount of time that it takes the electrical signals that represent bits to travel down the physical wire. Optical cable sends bits at about ~5.5 µs/km, copper cable sends it at ~5.606 µs/km, and satellite sends bits at ~3.3 µs/km. (There are a few additional microseconds of delay from amplifying repeaters in optical cable, but compared to distance, the delay is negligible.)

Distance delay can have a significant impact on application performance for applications that require a large number of network round trips in order to complete a transaction – for example, custom transactional based applications, database queries, and VoIP, which begins do degrade when one-way end-to-end latency exceeds 200-220 milliseconds.

One of the biggest sources of end-user ire are database queries designed to run over a LAN ported to the WAN. For example if a user executes a SQL database query that requests 100 rows of a database table, one row at a time, over a link with a latency due to distance of 60 ms, it would take approximately 6 seconds (60 ms * 100 turns) to complete the transaction. The same query executed by a user on a LAN connected to the same database server would take less than 2-3 ms to be completed, as the latency due to distance across the LAN is insignificant.

Queue Delay is the amount of time a packet must spend in a network buffer waiting its turn to be transmitted. Network interfaces transmit one frame at a time, typically one bit at a time. As such, when two or more packets are forwarded to a network interface at the same time, or close to the same time – one packet is transmitted while the others are put in a queue on the interface buffer to await their turn at the interface. Packets that are put into the queue must wait until they can be transmitted, adding milliseconds of delay.

Increases in Queue Delay can be measured and detected by monitoring traffic along a given network path. Typically, most intermittent increases in latency above the baseline distance latency can be attributed to network congestion. (In order to reduce the possibility of excessive queue delay, application servers that are members of the same application architecture should be placed on the same Ethernet switch and on the same VLAN to ensure they do not have to compete for uplink bandwidth when problems like the one pictured above occur.)

Worse still, if the problem gets worse and packets wait in increasingly longer lines within the queue, the buffer may become full and the packets may be dropped. Packet drop, in turn, causes TCP connections to throttle back on the rate of transmission.

Those are some of the main causes of latency – but what about jitter?

Jitter is a term that refers to the variance in the arrival rate of packets from the same data flow, and abnormal jitter values can negatively impact real-time applications like VoIP and video. Jitter is typically created by three different mechanisms in a network: variance in Serialization Delays due to variance in packet sizes, variance in per-packet Queue Delay due to packet spacing from multiple sources at a common outbound interface, or packets taking different routes from source to destination – perhaps due to per-packet load sharing or routing issues.

The most effective way to deal with jitter is by using low-latency queuing for VoIP and video traffic on network interfaces with large serialization and/or queue delays. In addition, endpoints (such as IP phones) can use jitter buffers or playout delay buffers in order to deliver received packets at a constant rate to the end consumer. These buffers are typically 30-50 ms in depth, and thus they attempt to manage jitter values within these values on any single one-way path. While these buffers technically add 30-50ms in latency, they significantly reduce jitter. Since human beings don’t start to notice latency in VoIP or VideoIP applications till it hits about 200ms, if latency can be kept to under 150 milliseconds, then jitter can be significantly reduced using this method.

By Ben Erwin

Even the WAN optimization vendors are starting to realize that WAN optimization’s utility is diminished without visibility into the network.

Riverbed’s recent announcement shows that they, (and other WAN optimization and acceleration vendors), are recognizing the dependency between better network management and successful WAN optimization projects. The partner vendors mentioned in the announcement all provide some level of network management capabilities, from network configuration to application delivery monitoring. These partnerships are part of a growing trend since network management capabilities are critical to understanding WAN optimization’s ROI and impact on network performance.

In addition to third-party partnerships, WAN optimization vendors have also developed their own technology to improve integration with network management vendors. Riverbed, Juniper, Cisco, and Blue Coat (Packeteer) have all developed traffic flow export capabilities within their WAN optimization appliances to help customers better understand WAN optimization’s impact on application flows. For example, a large insurance brokerage company exports traffic flow records from Riverbed appliances to NetQoS ReporterAnalyzer to help visualize changes to volume, rate, and bandwidth utilization for every application on the optimized link. This flow export alongside ReporterAnalyzer provides the customer with continued visibility across the link for future troubleshooting, traffic analysis, or capacity planning efforts.

While traffic flow and per application bandwidth utilization information is critical to managing application delivery, it’s only part of the story. The other part is measuring latency of mission critical applications between remote sites and the data center – a more tangible metric of WAN optimization’s ROI and impact on the end-user experience. This measurement can be nearly impossible to obtain since WAN optimization appliances obfuscate application transactions between clients and servers, breaking up the TCP stream.

In order to get around the broken TCP stream problem, we at NetQoS entered into a partnership with Cisco to provide unique technology which measures end-to-end latency over Cisco Wide Area Application Services (WAAS) optimized WAN connections. By integrating our NetQoS SuperAgent technology, WAAS users can get client and server-side data collection and reporting capabilities.

To our knowledge, Cisco and NetQoS currently provide the market’s only solution for accurate latency measurements for client and server communication over optimized links.

WAN optimization is all about improving application delivery. Collecting volume, rate, and bandwidth utilization on optimized applications is only part of the solution. Truly understanding ROI on WAN optimization requires accurately measuring network latency and the end-user experience.

Ben Erwin is technical marketing manager at NetQoS and on Tuesday, May 27, 2008, he will be presenting a Webinar on Building Performance-first Application Delivery Networks with Cisco and NetQoS.

Kevin Davis, a senior consultant at NetQoS, will be presenting a few training sessions at Symposium about SuperAgent, the end-to-end response time module of the NetQoS Performance Center. This will include a training session about how to use time-based network metrics in troubleshooting.  He talks about his upcoming training session below.

In the session, I’m going to be covering the importance of using a time-based metric in troubleshooting, because end-users complain foremost about time.  For example, they’ll say “the application is running slow,” or they believe “the network is slow.”  To users, everything is based on time, that’s what they’re complaining about.  And they’re correct.

It’s very new to many people to think of performance in “time” although that may seem counterintuitive - because most people are used to reading utilization graphs.  With utilization graphs, however, we don’t know if 70 or 80 or 90 percent utilization is necessarily impacting the user experience.  I mean, we buy networking equipment, routers, switches, firewalls, servers, and we want them to be highly – or efficiently - utilized.  Seeing high utilization could indicate a problem – or it could just indicate that you haven’t over-purchased.  So you can have a link at 90% utilization or a router at ninety percent CPU utilization but you won’t know if that’s impacting the end-user without a time based metric.

It’s time-based data that tells you how the users are being impacted.  Sure, the utilization data – the interface utilization, memory utilization, I/O utilization, can often tell what is doing the impact.  But the time base shows you the degree of the impact – the real-world effect on end-users.  With a time-based instrument, such as NetQoS SuperAgent, you can find out where the delay increase is occurring, and whether it’s based in the network, server, or application. 

In fact, you can take a look at time-based data and make a determination very quickly as to which entity is creating the performance issue – the beautiful thing about SuperAgent, in particular, is that it trends by time 24/7, so not only can you determine how your important business applications are being impacted today, but you can go back and look at recurring patterns in performance issues.  You can see if today is worse than yesterday or last week or last month.

In the session, I’ll also be going over how to architect the data center for performance.  Placement of servers that participate in inter-architectures is critical for the health and performance of the application and indeed the data center.  We also talk about how different protocols, for example, Microsoft’s TCP/IP stack, can impact application performance by enhancing or degrading it. 

It’s important for servers that are serving the same application.  For example, a front-end Web server and a back-end Oracle database really should be on the same switch on the same VLAN.  That way they receive optimum service from the network.  If they do leave the switch, they’ll have to contend with bandwidth going up and down the switch links, and they’ll be switched and routed multiple times. 

Based on measurements from customer environments and from our own laboratories, when two servers are on different switches they can have up to 18 milliseconds delay between them.  If we think of that in the terms of network engineers of one millisecond per 100 miles, what in effect we’re doing when we put two different servers on different switches, or two different VLANs on the same switch, we’re making it look like those servers are 1800 miles apart – like one server is in Los Angeles and the other is in Memphis. 

Gerald Combs is the original author and lead developer of the open-source, multi-platform, Wireshark network protocol analyzer. Combs works for CACE Technologies – a company which makes products that compliment Wireshark.  Today he mostly takes care of the administrative parts of the project but still does development as well, and he controls the version numbers and release schedule.

After ten years of development, Wireshark finally reached the milestone of a 1.0 release.  We speak to Mr. Combs in an interview below: 

NPD: So what is Wireshark?

Combs: Wireshark is a network protocol analyzer.  It’s kind of a traditional analyzer in that it’s a GUI that has three panes, the top pane shows a list of the packets captured off the wire, the middle pane a detail of whatever packet you have selected, and the bottom page shows the actual hex output – the bytes in the actual output.

NPD: Why did you decided to build Wireshark?

Combs: Years ago, I worked at a small ISP in the Midwest, and unfortunately, they couldn’t get me a Sniffer, which was the standard analyzer at the time, and of the products out there that were available, none of them ran on the platforms we used at the ISP – namely Solaris and Linux.  So I decided to sit down one day and start writing my own analyzer. 

I did the first release in July of 1998, and soon after started getting a steady stream of contributions from a bunch of really smart people.  After that, we built up a pretty good following of users.  And right now, Wireshark is the world’s most popular network protocol analyzer. 

NPD: Why did you decide to open-source the project?

Combs: I’d used open source software for a long time at that point.  Before then, I worked at a university and we made a lot of use of open source software.  It just made sense to me.  I wanted to give back to the community and it just seemed like a good way to go.  As it turned out, it was a great way to go, because Wireshark is appealing for a lot of people who write code for it.

NPD: Why has it taken ten years to reach Version 1.0? 

Combs: I just wasn’t comfortable until recently putting out the 1.0 release.  I’ve known for years - shortly after we made the initial release, people started using it in production environments.  And some people had trepidation because it wasn’t 1.0 yet.  But for the most part, people just didn’t care about the version number and they used it wherever they wanted to and wherever they needed to. 

But for me there were a number of features that were crucial and missing until recently that prevented me from putting a 1.0 stamp on it.  Probably the last one, one of the main ones, was privileged operation on Linux – getting it so that you could capture as root but run the GUI as non-root user. 

NPD: Have people come up to you and told you about how Wireshark helped them out?

Combs: I get e-mails from time to time from people, saying that I’ve helped them out.  I have some former coworkers that have mentioned that.  It’s actually pretty encouraging. 

We get a huge amount of code each month.  Between each release, we have 200,000 and a million lines of changes.  That’s just changes.  The actual source code is about 1.5 million lines now.  That’s a bigger job than I can do individually.  And luckily there are a bunch of smart and talented people that can help me with that.

NPD: What was the greatest challenge in developing Wireshark?

Combs: The greatest challenge is just the day-to-day maintenance, keeping the project going.  But several years ago, we had an initial push of fixing security bugs a while back and it was a huge undertaking. I just remember spending several months doing nothing but fixing these security bugs, and it was a big chore.   We have a huge codebase now, and unfortunately we just don’t have the resources to audit that.  But we have a lot of automated processes in place like fuzzing and static analysis that help us find those bugs. 

I can’t say this enough: Thank you to all the Wireshark developers out there and the user team – this has just been a great journey and it’s one that I hope to continue. 

by Patrick Ancipink
Director of Product Marketing, NetQoS

There’s been a lot of growth (and attendant hype) in technology areas like WAN optimization and application acceleration over the past few years, and for good reason. Anything that helps companies speed up and reduce the risk of strategic IT initiatives like consolidating data centers, turning up new branches or serving an increasingly mobile and scattered user community will be popular.

To help with cope with the increasing reliance on the WAN and keep latency in check, there are a dizzying array of vendors and products out there – but if you’re trying to determine precisely which techniques and technologies to implement for your specific needs, the array of vendors quickly goes from “dizzying” to “disorienting” and finally “nauseating.” 

Cisco’s been in this Tilt-a-Whirl™ of a market for a while (and NetQoS has been right there with them) and they’ve taken some big steps recently to provide a more holistic approach that centers on building an “application aware” network, rather than trying to highlight one type of implementation against another for a narrow set of capabilities.

NetQoS started working exclusively with Cisco closely to help customers evaluate, measure, and prove the effectiveness of WAN optimization and application acceleration deployments. As customers are moving from pilot phases into full production, the before/after measurements and comprehensive monitoring are critical to ensure customers are getting the benefits they intended and doing what they need to deliver application performance. 

To help get the word out, Cisco just launched a new section of their web site today that contains a wealth of information about, as they call it, “WAN and Application Optimization.” The downloadable presentation, Cisco WAN and Application Optimization Technical Overview Presentation, puts Cisco technologies (and complimentary ones, NetQoS included) into a useful context with a methodical approach and framework built around four steps: Profile and Baseline, Optimize, Evolve, and Operate. A whole Campbell’s Factory of Cisco alphabet soup technologies are included—WAAS, ACE, NBAR, Netflow, CBQoS, IP SLA, PfR—to show how they work in concert and what role they play in the bigger picture.

There’s also the Cisco WAN and Application Optimization Solution Guide , a very in-depth publication—like 227 pages deep—that is targeted for “technical personnel involved in the specification, design, and implementation of specific WAN and application optimization solutions.” We, here at NetQoS, are proud to have contributed several sections to book regarding the methodology and implementation of network performance monitoring for WAN optimization and application acceleration. 

(If you are looking for some lighter fare, the video on the site tells a nice story in about 6 minutes including an airshow, snowmobiles, windsurfers, and skydiving—interesting choices for demonstrating the criticality of serving video over the WAN.  Then again, some company somewhere has to make the recreational products, I suppose.)

For open source projects, v1.0 is generally a major milestone; one that is usually well earned.  After all, in open-source software, changes between versions are incremental and it can be a long time before hitting the 1.0 milestone.  For example, Mozilla – the original, before Firefox eclipsed it and it became SeaMonkey - spent four years as “beta” versions before finally getting the 1.0 designation – and it is notable that most of those beta versions were quite usable, Like much open-source software, it kept improving but just didn’t meet the developers exacting standards for a 1.0 release until it passed a threshold.

This 1.0 barrier was just reached for WireShark, the open-source packet sniffer. 

Is it a milestone?  Perhaps it’s just the ticking of the odometer over into the 1.0 area – the changes from the previous pre-1.0 version were minor – a new experimental version on MacOSX Intel, and some security related vulnerabilities patched.  However, WireShark remains a invaluable tool for anyone working in the network space.  Bill Alderson uses Wireshark for monitoring application performance, TCP behavior, retransmission symptoms, and protocol incongruities in situations in collaboration with NetQoS’s products – or where NetQoS’s products would be considered overkill.

Wireshark v1.0 is especially important because of its free-as-in-beer-ness.  Wireshark opened up network and application troubleshooting to rank-and-file IT staff – as anyone could download and use it, rather than having to wait for the network engineer to show up on-site or waiting for a third party.  Anything that helps the IT group solve problems faster is a good idea. 

Right now, Joel Trammell, our CEO, is at Sharkfest 08 (the first Sharkfest) at Foothill College, where in addition to Gerald Combs, the original developer of Wireshark, Dr. Vinton Cerf will be delivering a keynote. 

Additionally, with April Fools Day coming up tomorrow, you have to appreciate any program that allows you to play harmless practical jokes – and details exactly how on the official wiki. 

By Ben Erwin

Thanks to the latest Die Hard movie, I'm still fighting the urge to unplug my microwave to foil hacker attempts. Thanks to the U.S. government, however, we have a new line of defense against kitchen appliances of mass destruction.

The U.S. government has setup a new command center in the Air Force called Air Force Cyber Command or AFCYBER. Here's the summarized mission of AFCYBER, according to Air Force Secretary Wynne:

"The aim is to develop a major command that stands alongside Air Force Space Command and Air Combat Command as the provider of forces that the President, combatant commanders and the American people can rely on for preserving the freedom of access and commerce, in air, space and now cyberspace."

There are real threats; Estonia came under attack from hackers back in April of 2007 And in September of 2007, the U.S. Defense Department said that the Chinese military hacked into a Pentagon computer network.

It's hard to tell exactly how much damage a hack into U.S. computers could do because the Pentagon isn't exactly forthcoming with information on this. A plausible scenario would be a Chinese hacker gaining knowledge about U.S. troop movement. (A much less plausible scenario would be a teenage hacker who is looking for a game company accidentally, through a back door left by a programmer who left the project years ago, activates an AI which then seeks global thermonuclear war under the pretense that it cannot distinguish between a gamed scenario and reality.)

From a network monitoring and management perspective, AFCYBER will bring a whole new level of opportunities and challenges. How exactly do you monitor the United States network? What is the United States network? There are obviously some critical assets (White House, Pentagon, Capitol, etc.), but how many "cyber security holes" exist between critical infrastructure and those who want to attack critical infrastructure? Don't we all share some connectivity medium at some level?

It gets even more interesting on the offense front. Are you confident enough in your network management/security monitoring tools to launch a missile attack on an offending host? False positives take on a whole new meaning.

If you have answers or insights, I'd love to hear them. Otherwise, I may never microwave again.

We're releasing NetQoS VoIP Monitor v1.1. Biggest changes: SIP (Session Initiation Protocol) support, automatic and on-demand problem investigation, and capacity planning reports.

I want to start with SIP support, because there's an interesting related story that caught my attention when it came out on Slashdot.

One of the odd things about SIP is that it is, to some extent, a peer-to peer based protocol. The advantage of this is that it only requires a simple core network, with all the fiddly bits distributed to the network edge. This makes SIP more scalable than other protocols. You can see why our customers think SIP support is important and why NetQoS worked to put it into this release.

But as a side effect of the way the technology was designed, SIP's peer to peer network means that it can be difficult to route emergency calls because of the mobility of IP endpoints and the fact that SIP has no network location capability - you'll remember that Vonage got into a little bit of trouble a while back because it couldn't consistently promise E911 support. (That has since been fixed.)

SIP also establishes a VoIP connection directly between the two calls out at the edge. Once the call is set up, the data does not pass through any sort of central server owned or controlled by the VoIP provider. That makes it harder for the government to legally (or whatever) intercept calls.

I mention this because the actual documents governing the rules behind U.S. government interception of VoIP was leaked to Wikileaks on the 15th of March. Now, this is nothing new - CALEA requires VoIP providers to maintain wiretapping capability - just like the plain-old-telephone-service providers are. It's interesting, however, to see the documents. Or at least it might be to somebody else who is interested in network security and encryption.

But from a performance angle, the CALEA requirements for wiretapping are directly in opposition to the efficiency of a SIP VoIP network - that is, if a service provider must be in the middle of every call, it eliminates the benefits of the P2P structure. That adds a lot of network overhead.

The other new features in VoIP Monitor v1.1 are generally less conversation sparking - but no less important. Most of our other products, such as SuperAgent, have both automatic and on-demand problem investigation and capacity planning reports. These capabilities have been added to VoIP monitor in the new version.

Automatic investigations occur when a VoIP performance threshold - such as delay to dial tone - is exceeded. Then VoIP Monitor traces the call signaling path and compares it to the automatically generated baselines.

As far as capacity planning reports helps go, VoIP Monitor v1.1 providers enhanced reports on call volume, call quality, call failures, grade of service and gateway utilization. It provides a view of the effect of different call volumes. With this information, IT professionals can view capacity for specific locations or gateways or for the enterprise as a whole - the utilization reports are especially useful when negotiating contracts with service providers.

We have a demo of VoIP Monitor v1.1 up and running at VoiceCon in Orlando, at booth #1305, if you're attending.

Verizon and NBC are working on serving up TV shows to home computers. The problem is, high definition video, (and I've done some HD video work for the Web - shameless plug), takes a whole mess of bandwidth.

Now, the obvious solution for NBC would be to move to some sort of peer-to-peer distribution system, right? I mean, it works for Linux distros.

The problem is that a normal peer-to-peer connection doesn't distinguish between the cheap local links - that is, links on the same ISP, in roughly the same geographic area - from the expensive remote links. So while P2P provides a more cost effective solution, it doesn't provide the most cost-effective solution for the ISP.

A third party, Pando, has developed a P2P system for pre-authorized, pre-approved content, and has come up with a way to force peer to peer connections to look for local nodes first. This increase the efficiency of the system, lowers the cost, and generally increases the performance of the streaming/downloading video.

This is exactly the type of thing we talk about when we say that how the application is coded can have a huge impact on the application performance over the WAN. Sometimes instead of needing more bandwidth, you need to find a way to make the apps work more efficiently.

In this case, decentralized P2P systems developed after the destruction of Napster. Though they were much less likely to get shut down by the RIAA, they were also much less efficient. This dominated development of P2P applications for years. But for offering only pre-authorized content, a centralized system - especially one that takes advantage of the structure of the physical network, makes a certain bit of sense.

NBC will be offering Verizon customers their shows via Pando's P2P service - which they're calling P4P, later this year. The name is a logical outgrowth, P2P, or "peer to peer," versus P4P, or "peer for peer." P3P was disregarded because it sounded too much like PCP. And if a kid with a lisp goes around school saying: "I downloaded the latesth Methallica album on P3P" and a teacher hears: "I downloaded the latest Metallica album on PCP," well, that's just not going to be a story that ends well, now, is it?

There's only one problem with Pando's plan: Each ISP will have to give up information about its subscribers in order to participate - that is, the Pando platform requires knowing which nodes are "local" and which nodes are "remote" in order to optimize for the local connections:

For other ISPs to reap the benefits Verizon did in the test, they too would have to share information about their networks with file-sharing companies, and that they normally keep that information close to their chests.

''That's one of the objectives we have to solve -- how are we going to consolidate this data and distribute it?'' Pasko said, adding that the result of the test gives ISPs plenty of incentive to collaborate.

(Okay, maybe there's two problems: No offense to NBC, but when your biggest hit is a veritable case study in game theory… you need some new shows.)