Ken Church, Albert Greenberg and James Hamilton of Microsoft recently put out a paper on “Delivering Embarrassingly Distributed Cloud Services.”[PDF] Like most papers of this type, it’s a dry read, but informative. It looks at the tradeoff between mega-data center size and micro-data center diversity from the both the viewpoints of total cost of ownership and of performance.

The most important line in the entire report, of course, is “The trade-offs vary by application.” However, they make the argument that applications with little need for server-to-server communications will show benefits in cost, scale, reliability and performance through geo-diversification – in other words, lots of little datacenters as opposed to one big datacenter.

This seems to fly in the face of the trend in data consolidation, but there is a point to it: For any data center, there needs to be redundancy, but in a centralized data center, there needs to be more redundancy than having multiple small data centers. As Church, Greenberg, and Hamilton put it, “the more geo-diversity, the better. N+1 redundancy becomes more attractive for large N.”

The part that really interested me, though, was the networking section. (Section 3, in case you want to skip right to it.) Church, Greenberg, and Hamilton point out that in a large, centralized datacenter, you can have end-to-end control and assure a particular level of performance through supported service level agreements. On the other hand, they argue:

“[with distributed data centers] the cloud service provider has ceded control of quality to its Internet access providers, and so cannot support (or even fully monitor) SLAs on flows that cross out multiple provider networks, as the bulk of the traffic will do. However, by artfully exploiting the diversity in choice of network providers and using performance sensitive global load balancing techniques, performance may not appreciably suffer. Moreover, by exploiting geo-diversity in design, there may be attendant gains in reducing latency…”

“Many large analysis applications are best run centrally in mega data centers… Interactive applications are best run near users… [they] can be delivered with better QoS (e.g., smaller TCP round trip times…) via micro data centers.”

The argument’s sound, especially when you consider that interactive applications are probably the most latency sensitive because they need to make multiple trips to and from the client and server with every interaction.

But reducing the propagation delay (or distance delay) is merely one part of the performance equation. By ceding control over router performance and transmission, you have no way of diagnosing network round trip time problems if they occur, and wouldn’t be able to fix them – short of the messy step of changing service providers – even if you did. If something goes wrong, it could negate the speed increases by diversifying servers, so moving to this model more of a gamble than a guarantee of improvement. Granted, it’s a gamble that might make sense for some apps and some organizations – some apps, apparently, can get away with less than 100% uptime.

Reprinted from TheDailyWTF.com:

At my company, there's a bit of a wall between Application Development and Network Operations. All "network and network-service related issues" must be reported through the porthole, a.k.a. Helpdesk. Quite often, this leads to interesting results.

"Helpdesk, Jerry speaking."

"Hey Jerry," I said, "this is Paul over in app dev. Our TerraTrade system has a defective ForEx feed that needs to be fixed right away. It's causing a bit of an outage, so if possible, can we open the ticket as 'Urgent'?"

"Not a problem," Jerry responded, "let me just have your name and number, and we'll take care of it."

I gave him a few more details and felt pretty happy that helpdesk was actually helpful. Five minutes after I hung up, an email message came in:

[URGENT] TICKET #71248 HAS BEEN ASSIGNED TO YOU

Not a moment later, my phone rang. I hesitantly picked it up.

"Hello is this Paul," the caller asked before I could even say Hello. I affirmed that it was me.

"Paul," he said, "this is Steve over at help desk. We've got an Urgent trouble ticket that just came in. It's for a, uh, Fortix feed defect. We just wanted to make sure you're on it?"

It took me a few minutes to explain to Steve that he was, in fact, assigning me the defect I had just reported.

Before you laugh too hard at the above story – it’s not that far removed from what many IT departments do daily – play the blame game. The user calls a problem into the help desk, then assigns it to the network. The network calls the helpdesk to tell them that it’s a problem with the server, and the server team calls the help desk to tell them it’s the application. If you’re lucky, someone along that chain will know how to fix the problem, but even if you are lucky, it’s still a lot of wasted time and energy.

This line from Manish Chacko’s article, “God Help the Help Desk” sums it up:

Imagine a man walking into a hospital, saying that he doesn't feel good, and doctors around the country are immediately called in, starting with the cardiologist, who rules out heart trouble. The man is next wheeled to a podiatrist, who rules out any problems with his feet. He's then wheeled to a gynecologist (But I'm a man... Ma'am, I'm a doctor. I think I should make that determination - and only after the tests come back.) If your diagnostic process is trial by error, you're not, technically, diagnosing.

This is why Dr. Jim Metzler recommended time and time again that application development and network operations merge into a single “application delivery” team. The primary job of IT is to deliver an application. Focusing on the performance your single group misses the point – it’s how the applications perform that is most important, and hardware, software, and networking are all part of that performance equation.

By Kevin Davis
Adapted from “Sources of Latency” Whitepaper

When network users call the Help Desk to report poor application performance, you don’t typically hear things like “The router’s CPU is too busy!,” “The network utilization is above 70%!,” or “The carrier path has failed-over to a sub-optimal path.” Instead, what you’re likely to hear is “The network is slow” or “The calls on my IP phone sound terrible.”

Complaints that end-users lodge are nearly always based their quality of experience using the application. And their quality of experience is almost always reliant on time.

Anytime a significant delay occurs in the delivery of network data, application performance suffers. Depending on the type of application and how it works, variances in network delay can have a severe impact on application performance thereby degrading end-user’s experiences.

Two important measurements of time intervals in network transmission systems are referred to as “latency” and “jitter”. Understanding latency and jitter sources and how their values vary in network architectures is critical to engineering application performance and optimizing information resources. For many regular readers, this will be old-hat, but we’ll go over it again.

Network latency is the amount of time it takes for a packet to be transmitted end-to-end across a network and is composed of five variables:

Network Latency = (Distance Delay) + (Serialization Delay) + (Queue Delay) + (Forwarding Delay) + (Protocol Delay)

Serialization Delay refers to the amount of time it takes for a network interface (such as a router’s interface or computer’s NIC) to perform bitwise transmission of a frame unto the outbound media, Forwarding Delay is the amount of time it takes a network device to process a frame/packet by performing a destination address lookup and forwarding the frame/packet to the outbound interface, and Protocol Delay is the amount of time that access or transmission algorithms may contribute to the delay of a network frame, and is typically introduced at the endpoints of the data transmission system.

Serialization delay, on a per-packet basis, becomes insignificant at data rates above 1.544 Mbits/s – or a T1. Forwarding delay is typically insignificant in modern routers and switches (when appropriately configured – significant delay can occur in misconfigured routers.) And Protocol delay typically occurs at the access layer or the end points. So the two major variables that have the most effect on network latency are Distance Delay and Queue Delay.

Distance Delay is simply the minimum amount of time that it takes the electrical signals that represent bits to travel down the physical wire. Optical cable sends bits at about ~5.5 µs/km, copper cable sends it at ~5.606 µs/km, and satellite sends bits at ~3.3 µs/km. (There are a few additional microseconds of delay from amplifying repeaters in optical cable, but compared to distance, the delay is negligible.)

Distance delay can have a significant impact on application performance for applications that require a large number of network round trips in order to complete a transaction – for example, custom transactional based applications, database queries, and VoIP, which begins do degrade when one-way end-to-end latency exceeds 200-220 milliseconds.

One of the biggest sources of end-user ire are database queries designed to run over a LAN ported to the WAN. For example if a user executes a SQL database query that requests 100 rows of a database table, one row at a time, over a link with a latency due to distance of 60 ms, it would take approximately 6 seconds (60 ms * 100 turns) to complete the transaction. The same query executed by a user on a LAN connected to the same database server would take less than 2-3 ms to be completed, as the latency due to distance across the LAN is insignificant.

Queue Delay is the amount of time a packet must spend in a network buffer waiting its turn to be transmitted. Network interfaces transmit one frame at a time, typically one bit at a time. As such, when two or more packets are forwarded to a network interface at the same time, or close to the same time – one packet is transmitted while the others are put in a queue on the interface buffer to await their turn at the interface. Packets that are put into the queue must wait until they can be transmitted, adding milliseconds of delay.

Increases in Queue Delay can be measured and detected by monitoring traffic along a given network path. Typically, most intermittent increases in latency above the baseline distance latency can be attributed to network congestion. (In order to reduce the possibility of excessive queue delay, application servers that are members of the same application architecture should be placed on the same Ethernet switch and on the same VLAN to ensure they do not have to compete for uplink bandwidth when problems like the one pictured above occur.)

Worse still, if the problem gets worse and packets wait in increasingly longer lines within the queue, the buffer may become full and the packets may be dropped. Packet drop, in turn, causes TCP connections to throttle back on the rate of transmission.

Those are some of the main causes of latency – but what about jitter?

Jitter is a term that refers to the variance in the arrival rate of packets from the same data flow, and abnormal jitter values can negatively impact real-time applications like VoIP and video. Jitter is typically created by three different mechanisms in a network: variance in Serialization Delays due to variance in packet sizes, variance in per-packet Queue Delay due to packet spacing from multiple sources at a common outbound interface, or packets taking different routes from source to destination – perhaps due to per-packet load sharing or routing issues.

The most effective way to deal with jitter is by using low-latency queuing for VoIP and video traffic on network interfaces with large serialization and/or queue delays. In addition, endpoints (such as IP phones) can use jitter buffers or playout delay buffers in order to deliver received packets at a constant rate to the end consumer. These buffers are typically 30-50 ms in depth, and thus they attempt to manage jitter values within these values on any single one-way path. While these buffers technically add 30-50ms in latency, they significantly reduce jitter. Since human beings don’t start to notice latency in VoIP or VideoIP applications till it hits about 200ms, if latency can be kept to under 150 milliseconds, then jitter can be significantly reduced using this method.

By Ben Erwin

Even the WAN optimization vendors are starting to realize that WAN optimization’s utility is diminished without visibility into the network.

Riverbed’s recent announcement shows that they, (and other WAN optimization and acceleration vendors), are recognizing the dependency between better network management and successful WAN optimization projects. The partner vendors mentioned in the announcement all provide some level of network management capabilities, from network configuration to application delivery monitoring. These partnerships are part of a growing trend since network management capabilities are critical to understanding WAN optimization’s ROI and impact on network performance.

In addition to third-party partnerships, WAN optimization vendors have also developed their own technology to improve integration with network management vendors. Riverbed, Juniper, Cisco, and Blue Coat (Packeteer) have all developed traffic flow export capabilities within their WAN optimization appliances to help customers better understand WAN optimization’s impact on application flows. For example, a large insurance brokerage company exports traffic flow records from Riverbed appliances to NetQoS ReporterAnalyzer to help visualize changes to volume, rate, and bandwidth utilization for every application on the optimized link. This flow export alongside ReporterAnalyzer provides the customer with continued visibility across the link for future troubleshooting, traffic analysis, or capacity planning efforts.

While traffic flow and per application bandwidth utilization information is critical to managing application delivery, it’s only part of the story. The other part is measuring latency of mission critical applications between remote sites and the data center – a more tangible metric of WAN optimization’s ROI and impact on the end-user experience. This measurement can be nearly impossible to obtain since WAN optimization appliances obfuscate application transactions between clients and servers, breaking up the TCP stream.

In order to get around the broken TCP stream problem, we at NetQoS entered into a partnership with Cisco to provide unique technology which measures end-to-end latency over Cisco Wide Area Application Services (WAAS) optimized WAN connections. By integrating our NetQoS SuperAgent technology, WAAS users can get client and server-side data collection and reporting capabilities.

To our knowledge, Cisco and NetQoS currently provide the market’s only solution for accurate latency measurements for client and server communication over optimized links.

WAN optimization is all about improving application delivery. Collecting volume, rate, and bandwidth utilization on optimized applications is only part of the solution. Truly understanding ROI on WAN optimization requires accurately measuring network latency and the end-user experience.

Ben Erwin is technical marketing manager at NetQoS and on Tuesday, May 27, 2008, he will be presenting a Webinar on Building Performance-first Application Delivery Networks with Cisco and NetQoS.

Kevin Davis, a senior consultant at NetQoS, will be presenting a few training sessions at Symposium about SuperAgent, the end-to-end response time module of the NetQoS Performance Center. This will include a training session about how to use time-based network metrics in troubleshooting.  He talks about his upcoming training session below.

In the session, I’m going to be covering the importance of using a time-based metric in troubleshooting, because end-users complain foremost about time.  For example, they’ll say “the application is running slow,” or they believe “the network is slow.”  To users, everything is based on time, that’s what they’re complaining about.  And they’re correct.

It’s very new to many people to think of performance in “time” although that may seem counterintuitive - because most people are used to reading utilization graphs.  With utilization graphs, however, we don’t know if 70 or 80 or 90 percent utilization is necessarily impacting the user experience.  I mean, we buy networking equipment, routers, switches, firewalls, servers, and we want them to be highly – or efficiently - utilized.  Seeing high utilization could indicate a problem – or it could just indicate that you haven’t over-purchased.  So you can have a link at 90% utilization or a router at ninety percent CPU utilization but you won’t know if that’s impacting the end-user without a time based metric.

It’s time-based data that tells you how the users are being impacted.  Sure, the utilization data – the interface utilization, memory utilization, I/O utilization, can often tell what is doing the impact.  But the time base shows you the degree of the impact – the real-world effect on end-users.  With a time-based instrument, such as NetQoS SuperAgent, you can find out where the delay increase is occurring, and whether it’s based in the network, server, or application. 

In fact, you can take a look at time-based data and make a determination very quickly as to which entity is creating the performance issue – the beautiful thing about SuperAgent, in particular, is that it trends by time 24/7, so not only can you determine how your important business applications are being impacted today, but you can go back and look at recurring patterns in performance issues.  You can see if today is worse than yesterday or last week or last month.

In the session, I’ll also be going over how to architect the data center for performance.  Placement of servers that participate in inter-architectures is critical for the health and performance of the application and indeed the data center.  We also talk about how different protocols, for example, Microsoft’s TCP/IP stack, can impact application performance by enhancing or degrading it. 

It’s important for servers that are serving the same application.  For example, a front-end Web server and a back-end Oracle database really should be on the same switch on the same VLAN.  That way they receive optimum service from the network.  If they do leave the switch, they’ll have to contend with bandwidth going up and down the switch links, and they’ll be switched and routed multiple times. 

Based on measurements from customer environments and from our own laboratories, when two servers are on different switches they can have up to 18 milliseconds delay between them.  If we think of that in the terms of network engineers of one millisecond per 100 miles, what in effect we’re doing when we put two different servers on different switches, or two different VLANs on the same switch, we’re making it look like those servers are 1800 miles apart – like one server is in Los Angeles and the other is in Memphis. 

Traffic shaping is not a tool of the devil, nor do we believe the solution to bandwidth problems is simply to provision more dark fiber and build more underground fiber optic lines. But as time has gone on, the issues around network neutrality have become more pronounced.

For example, Bell Canada has been throttling P2P service, much like Comcast in the United States. However, what makes this different is that Bell Canada is in a position much like AT&T – in that throttling the network on the backbone affects all the people – including people who are not Bell’s customers – along the line.

Worse still, Bell has been reselling the capacity to provide ADSL service to smaller ISPs without letting the services know that the bandwidth is throttled for certain applications. One of those smaller ISPs, Teksavvy, said: “We are not throttling anything and as far as I am aware will never throttle anyone. We don't believe in it.”– so the idea that Bell will leave them with no choice in the matter is a little worrisome. There isn’t much choice in the matter – the only other big broadband provider in Canada is Rogers Cable, which also throttles traffic.

There are arguments that “net neutrality” will be solved by the forces of the free market – that is, if one ISP throttles, they can go to their competitors. The problem is that, in this case, this is exactly what savvy customers were doing by moving from larger companies, like Bell and Rogers, to smaller companies like Teksavvy. From the consumer, it’s reducing their choice. For the small ISP, it’s could be considered downright anticompetitive, and the Canadian Association of Internet Providers applied for relief before the Canadian Radio-television and Telecommunications Commission that would require Bell Canada to cease and desist.

We contacted Rocky Gaudrault, CEO of Teksavvy Solutions, but because this was now a legal matter, he explained that he was unable to comment. It was clear that he is passionate about the issue, but Teksavvy’s staff keeps him from speaking out by supplying him with timbits and beer to keep his mouth and hands busy.

Particularly interesting is this comment by a Slashdotter – both Bell and Teksavvy charge on a “tiered and metered” basis – which pretty much cuts through the false choice between deep packet inspection and metered bandwidth; Bell has both. (One profanity-laden post implied that the only reason that Bell Canada did this was to coldly eliminate the most compelling competitive advantage that smaller ISPs had – Bell had throttled traffic, small ISPs didn’t.)

The upshot is that network neutrality concerns have been brought to Canada’s Parliament during Question Time. (link via Prof. Michael Geist at the University of Ottowa, who we hope to have an interview with on Monday.) It’s unsurprising because these matters do not just affect consumers but large enterprises as well - an unannounced and sudden change in the QoS policies of the backbone provider is exactly the type of thing that can foul up capacity planning, VoIP switchover, teleconferencing, etc. Especially worrisome are those technology companies who rely on some form or another of P2P traffic to help cut their bandwidth costs.

Deep packet inspection is a powerful tool, and used in the right hands, in the right way, it can help make QoS planning easier, can help streamline business critical applications, can provide overall better end-user response times, and may indeed be a great technological boon.

But we can’t see any benefit in this case for throttling the traffic of resold bandwidth, and for not disclosing the changes in advance. If businesses that control backbone traffic want to avoid governmental regulation, they need to show that they can be responsible with the power they have and use it in a manner which is neither anti-competitive nor deceptive to wholesale resellers and end-user customers.

by Brian "Scrabble" Boyko
Editor, Network Performance Daily

When a company like Cisco goes into "new media marketing," it doesn't mess around. To promote the Cisco ASR 1000 WAN-edge router, it started a Facebook Group, a Second Life Site, and a slew of holiday mascot viral videos. But that's not the big one.

It won't win any praise from Ben "Yahtzee" Crowshaw, but Cisco created an entire video game around the router. The game, implemented in Shockwave, pits you as the lead agent, piloting an ASR1000 router - yes, piloting - across cyberspace, picking up packets according to quality of service priority, and delivering them across the network. There are also bonuses related to the router's capabilities, such as a 'throughput upgrade' that increases the speed of your… uh… "hover-router," and a "parallel processing" upgrade that allows you to pick up two different color balls - I'm sorry, I mean two different types of network traffic packets - instead of having to clear the packets from the board one color at a time. You might expect that there might be lasers or something coming out of the 'routercraft' but it's a router, so it doesn't have any lasers.

It shows that Cisco can have a little bit of fun with itself, and doesn't mind others poking fun at it either, otherwise they never would have put this out there for people like myself to poke fun at. But, as a game, it's amusing for five minutes, and certainly a great way to justify playing a video game at work, but if I'm going to be playing a video game at work, I'd rather play a game where I didn't actually learn anything about routers. There's a reason that despite the obvious pun, the Valve game developers didn't have GLaDOS go on about the relative merits of different firewall solutions as she tried to incinerate you.

The game itself may miss a few marketing targets, for example, the "space router" was frustrating to steer, even after the power-ups it was kind of sluggish, and it would frequently get rammed into the walls. Sure, I'm remembering the parallel processing thing as it's a great way to illustrate that particular feature, but I also remember a frustrating box that ran slowly and crashed repeatedly. I don't say this to knock at the router it is supposed to represent, but merely to knock the representation. Then again, the real ASR1000 can't fly around the room like a robot Peter Pan, so it's kind of a wash.

At any rate, the game is simple, amusing, and illustrates the main points, which is about as well as you can expect from a marketing mini-game, and hey, I'm talking about it and you're listening, so it can't have failed that badly.

Windows Server 2008 officially launched today with little fanfare; but the new enterprise-class operating system has been eagerly awaited by people who eagerly await operating systems, instead of going out and having a good time with their lives.

NetworkWorld has a thorough review of the W2K8 OS up on their site, but spends a bit of time tracking the performance of the network input output in various tests.

We tested network I/O performance using both emulated I/O and various traffic/assault tests (see How we did it) and found Windows 2008 Server performance has improved - and especially improved when Vista is the client….

The new stacks also have the ability to dynamically respond to communications latency in network connections as they possess the ability to dynamically change TCP packet window size, which allows a communication channel to be more efficiently stuffed with data.

In our testing we found that under light loads, the effects in terms of speed of tasks like copying folders, streaming media and loading complex Web pages aren't strongly demonstrated, but the effects under heavy loads, however, favors performance for Vista, strongly. Depending on the mixture of I/O (but pronounced under streaming media and heavy file copying), Vista can be as much as 43% faster than Windows XP SP2 in copying operations and 18% faster in opening concurrent streams.

This also means that there's a two-class affinity for clients of Windows 2008 Server Editions - Vista and everyone else, including Windows XP SP2, MacOS (we used 10.4.10 and 10.5.2) or other SAMBA clients that use SAMBA 3.0.2+ connection methods. If you have a client with the new stack, you're more efficient, and, therefore faster under higher loads, but you're a second-class citizen if your stack isn't up to date.

What I'd like to know is what, specifically, makes W2K8-server/Vista-client combinations so powerful. Is it just the compound TCP protocol? Are there kernel optimizations for network data processing? (I don't have the technical knowledge to address those questions, I'm hoping that my readers will be able to share their theories and the results of any tests they may run.)

At any rate, while W2K8 is a significant milestone release, good or ill, the history of server software distribution usually means a slow rollout period - to the point where naming your operating system by year becomes almost a bitter irony; chances are most companies who use W2K3 will want to roll out W2K8 in 2009 at the earliest.

By Brian Boyko
Editor, Network Performance Daily

Since yesterday's Network Performance Daily post which criticized Nicholas Carr for a quote in Network World, I've finished reading The Big Switch: Rewiring the World, from Edison to Google.

Keep in mind, I agree with the main thrust of Carr's arguments in The Big Switch and recommend it. The main thrust being that the software applications that were once developed in-house in a client-server model are increasingly moving towards "the cloud" of SAAS and Web applications.

The Network World article take on the book made it seem like Carr's core message was that IT departments, as we know them, would be obsolete. Admittedly, he does try to make that point in the book. I, however, don't think that that is the focus of the book.

There was one example from The Big Switch that stuck with me - partially because I'm a gaming geek as well as a techie. On page 114, Carr mentions "Second Life", a computer game which is mostly delivered as a service.

"Second Life is an example of a utility service supplied over the Internet and shared simultaneously by many people. It's very different from traditional computer games, which need to be installed separately on each player's hard drive. But Second Life is also itself a construction of many other utility services. The "computer" that runs Second Life doesn't exist in any one place; it's assembled, on the fly, from various data-storage and data processing molecules floating around in the global computing cloud… The program constantly "talks," over the Internet, with the main software Linden Lap uses to generate its online world. That software runs on hundreds of server computers that are housed in two data centers, one in San Francisco and one in Dallas, owned not by Linden Lab but by utility hosting companies. Every server computer contains, in turn, four virtual computers, each of which controls a 16-acre plot of land in Second Life. All the real and virtual computers work in tandem to create the vast world that residents experience as they play the game."

Second Life is an excellent example of "Gaming As A Service." There's just one problem with Second Life (other than the fact that most people don't even have enough time for their first life), and that's network performance.

The key draw of Second Life is that the world is entirely created by end-users. All attractions, games, and objects are the result of savvy Second Life end-users who have created these things to share or sell with other end-users. Unlike other MMORPGs, the world of Second Life is infinitely customizable, so it would be a bad idea to try to run it as a client-server application. Since all the information changes every time you play, (and sometimes while you play,) running Second Life as a service makes sense.

But there are significant drawbacks to this model. Loading up the information needed to get the details about the world, even on the fastest of Internet connections, takes forever. It's a bandwidth hog. Even if network performance conditions are ideal, rendering textures and shapes over the Internet is a time-consuming endeavor, and there is a very clear tradeoff between the quality of the visual application, and the quality of game's application performance. Controls aren't very responsive at all, mostly because the information about avatar movement is competing with graphics and world information on the pipe. This gives everything a frustrating, "bouncy" quality. Comparing that to a traditional client/server model type game - say, "World of Warcraft" - and the difference is apparent. WoW is quick and responsive, can handle multiple, and very complex, users very well, and while there may be lag on WoW at times, it never approaches the same amount of lag in Second Life. Even "Guild Wars," which has a 101KB client but is similar in scope, complexity, and gameplay as WoW, downloads the game software to the client at run time and caches it for the future rather than try to run the entire game off the server - and you can tell from the performance difference that, for right now, most gaming will continue to follow the client/server model.

Indeed, application and network performance is so important to gamers that even in an age where you can find a game of "Team Fortress 2," "Battlefield 2142" or "Quake Wars" any time, any place, 24 hours a day over the Internet, gamers lug their desktop systems around with them, get together with anywhere from 4 to 300 friends, connect it all to a single, created network, and play in what are known as "LAN parties." Why? Because there's less network latency, and better performance under a network that you control than there ever will be over even the best case online scenarios.

So will IT departments become obsolete? No - forgetting for a moment that somebody has to manage the infrastructure on the business side to allow all those end-users to connect to the cloud to access SAAS apps and "virtual data centers" and the like, the popularity of both World of Warcraft and Second Life show that both SAAS apps and client-server apps will be around, each model used for the advantages they provide in the cases where those advantages are beneficial.

It's breathtaking what is going on in the industry in this area and Carr puts his finger right on it in The Big Switch. There are revolutionary things going on in the SAAS field. Google Gears is bringing online apps offline. Virtualization is turning hardware into software, and when you turn hardware into software, you can offer "hardware-as-a-service."

But as for IT, well, maybe some companies will be able to make do - or be willing to risk - exclusively using SAAS solutions. But for most large companies, they need performance and control, not necessarily utility-like convenience, from their applications. IT departments aren't going anywhere anytime soon.

What do you think will be the future of IT in the SAAS environment? Feel free to discuss it in our comments section.

Julie Bort interviewed George Kurian at Cisco in Network World, where they talk about WAN optimization.

The interview talks about how Cisco's optimization and acceleration products are distinguished from competitors' and (of course, considering that George Kurian works for Cisco) promoted as superior because of their transparent placement in the network. This means they can be shared among several servers and applications, as well as integrated with Cisco's existing products, and QoS and security policies do not have to be migrated or disrupted. One item only barely touched upon is the idea of using a single appliance in the branch office - the Integrated Service Router - to handle WAN optimization, security, and routing - and how having one appliance to handle all these tasks helps cut down on server room clutter and complexity.

To be sure, these appear to be advantages to Cisco's solution. But the dirty little secret is that all WAN optimization solutions on the market, including Cisco's, obscure end-to-end performance metrics. This is a major issue, of course, and makes the current state of choosing whether or not to deploy a WAN optimization solution a Monty Hall problem - do you opt to retain visibility into your network performance and the ability to solve problems faster, or you deploy a WAN optimization device and hope that whatever's behind curtain number two (the resulting performance gain) is better than what you've traded for?

Maintaining transparency of response time and latency metrics is critical in our view and any WAN optimization vendor that provides a solution to this problem will have a serious competitive advantage.