Ken Church, Albert Greenberg and James Hamilton of Microsoft recently put out a paper on “Delivering Embarrassingly Distributed Cloud Services.”[PDF] Like most papers of this type, it’s a dry read, but informative. It looks at the tradeoff between mega-data center size and micro-data center diversity from the both the viewpoints of total cost of ownership and of performance.

The most important line in the entire report, of course, is “The trade-offs vary by application.” However, they make the argument that applications with little need for server-to-server communications will show benefits in cost, scale, reliability and performance through geo-diversification – in other words, lots of little datacenters as opposed to one big datacenter.

This seems to fly in the face of the trend in data consolidation, but there is a point to it: For any data center, there needs to be redundancy, but in a centralized data center, there needs to be more redundancy than having multiple small data centers. As Church, Greenberg, and Hamilton put it, “the more geo-diversity, the better. N+1 redundancy becomes more attractive for large N.”

The part that really interested me, though, was the networking section. (Section 3, in case you want to skip right to it.) Church, Greenberg, and Hamilton point out that in a large, centralized datacenter, you can have end-to-end control and assure a particular level of performance through supported service level agreements. On the other hand, they argue:

“[with distributed data centers] the cloud service provider has ceded control of quality to its Internet access providers, and so cannot support (or even fully monitor) SLAs on flows that cross out multiple provider networks, as the bulk of the traffic will do. However, by artfully exploiting the diversity in choice of network providers and using performance sensitive global load balancing techniques, performance may not appreciably suffer. Moreover, by exploiting geo-diversity in design, there may be attendant gains in reducing latency…”

“Many large analysis applications are best run centrally in mega data centers… Interactive applications are best run near users… [they] can be delivered with better QoS (e.g., smaller TCP round trip times…) via micro data centers.”

The argument’s sound, especially when you consider that interactive applications are probably the most latency sensitive because they need to make multiple trips to and from the client and server with every interaction.

But reducing the propagation delay (or distance delay) is merely one part of the performance equation. By ceding control over router performance and transmission, you have no way of diagnosing network round trip time problems if they occur, and wouldn’t be able to fix them – short of the messy step of changing service providers – even if you did. If something goes wrong, it could negate the speed increases by diversifying servers, so moving to this model more of a gamble than a guarantee of improvement. Granted, it’s a gamble that might make sense for some apps and some organizations – some apps, apparently, can get away with less than 100% uptime.

Cisco has put up a new video in their “Seminar and Webcast Series” talking about “Energy Efficiency in the Data Center.” It may be produced by Cisco but the key points are pretty much vendor-neutral – starting with the idea that “Green” computing is a political/PR buzzword, and the way enterprises should look at the problem is one of efficiency and of sustainability.

Data center power consumption has more than doubled since 2001; the worry is that the trend will continue on an exponential pattern. This power consumption mainly comes from cooling the servers, rather than powering the servers; and with each 1U server (running 24/7/365) requiring the same amount of energy per year as it would take a Toyota Camry to drive 15,000 miles, energy efficiency is crucial.

Part of the solution is to buy more efficient components that cost more up front but pay money back. Another part of the solution is virtualizing servers, consolidating servers, and decommissioning servers.

They also mentioned using provided utilities to step-down the voltage if the server was underutilized – a trick laptop owners have been doing to get more life out of their batteries on the road. Same concept – if you don’t need all the power, consume less of it.

As far as the network goes, data center consolidation brought on by advances in WAN optimization is a big step towards reducing utility costs. Another step is taking advantage of the movement towards putting tools in the network infrastructure itself rather than as separate appliances – for example, putting SuperAgent network monitoring software (shameless plug) into Cisco’s WAAS.

These are all some common sense solutions and probably not the first time you’ve heard them. But the key point of the video-seminar was that just as we keep harping on the fact that you need to baseline your network performance to ensure that the changes you make to your network are having the desired effect, you also need to baseline your power costs as you make improvements.

Recently, there’s been some discussion on Slashdot regarding MySQL in the past few months, after MySQL (the company) was bought out by Sun Microsystems.  MySQL (the company) has announced that they will be developing some proprietary add-ons to the backup capabilities of MySQL (the database) which will only be available to MySQL’s (the company’s) customers of MySQL (the database) enterprise edition, and not to MySQL (the database) community edition. 

This has been blown a bit out of proportion.  (The headline, on Slashdot, “Sun may begin close-sourcing MySQL” was misleading at best).  We e-mailed Steve Curry at MySQL (the company) and he pointed us to some information clearing up the situation.

· Anything that has been released as open-source under GPL continues to be released as open-source under GPL. Sun and MySQL (the company) are not going to start “closing” the open-source MySQL (the database,) and it seems unlikely that they will be able to legally do so even if they wished to.

· Improved backup capabilities are being planned in MySQL (the database) 6.0 for both the open-source community and open-source with proprietary add-ons enterprise version. 

· Proprietary add-ons are being added to the Enterprise version of MySQL (the database).  These add-ons are not core critical, they are essentially added-value for paying customers, which add compression, encryption, specific native drivers – things that a particular business might need but which aren’t critical to the core functioning of MySQL (the database.) 

· The decision to do so was done before MySQL (the company) was acquired by Sun Microsystems. If anything, Sun has been very open-source friendly, with Star Office forming the basis of OpenOffice.org, and Solaris and Java both open-source now.

· There is nothing preventing people from forking the MySQL (the database) source code and producing open-source versions of the proprietary capabilities.

The use of proprietary add-ons to an open-source system isn’t even all that rare.  Click N’ Run for Linux systems adds proprietary software to the open-source Linux; MacOSX is based on the BSD-licensed Darwin, a BSD-like distribution.

We also note the irony of a number of proprietary Web applications running off of LAMP stacks, where the L, the A, the M (the DB) and the P are all “free software.” 

There are a number of proprietary Web applications running with MySQL (the database) – and a move to “close source” MySQL (the database) would have messed with the business models of many companies – including NetQoS.  NetQoS uses MySQL (the database) Enterprise edition in our network monitoring and reporting products and we’re customers of MySQL (the company).  So we’re glad this whole thing is a tempest in a teapot. 

I tried to think of a prominent case where someone successfully “closed the source” of a flagship product after it was open-sourced - but couldn't until I went much, much farther afield.  There is a company “closing the source” on its major flagship product.

That company is Wizards of the Coast, a subsidiary of Hasbro.  And the flagship product is “Dungeons and Dragons.” 

Wizards (the company) makes Dungeons and Dragons, a role-playing, computer-less tabletop game where you play knights, elves, and powerful wizards (the characters) – a game that has a history of being very attractive to the technology-oriented crowd because of our love of math and power fantasies.

What makes Dungeons and Dragons particularly interesting is that a while back, Wizards (the company) released an “Open Gaming License” (OGL) which allowed third parties to develop additional content for Dunegons and Dragons, and, in fact, create entirely new games in different settings and genres using the rules established in Dungeons and Dragons 3^rd edition.  If you were a third-party company, you could publish supplements to provide traps, monsters, or new spells for wizards (the characters) to cast.  And many did.

This had numerous benefits all around; players needed to learn how to use only one system, and they had tons of D&D supplements to choose from, game companies found they had an audience in D&D players that they might not have otherwise had, Wizards (the company) found a sea of “developers” for their system which made ownership of D&D’s “core books” more valuable, and while it may not have resulted in a rebirth of the roleplaying game industry, it sure propped it up for a little while longer.

Because game players only had to learn one set of rules to play, the roleplaying game industry standardized quite a bit and the system used in Dungeons or Dragons (known as “d20”) became quite widely used, dominating the RPG field for a time. 

D&D “version 4.0” will soon be released, and many game beta testers believe the system has been radically overhauled and improved.  However, this new system will not be released under the OGL.  It will however, be released under the “Dungeons and Dragons 4^th Edition Game System License” (GSL). 

The GSL license has not yet been made public, but there are rumors, speculations, and concerns, fueled by online posts made by the brand manager and licensing manager for Dungeons and Dragons, and relayed by the lead writer of third-party publisher Necromancer Games that the GSL will contain a “poison pill” clause – that is, in order to use the GSL, a game company must not publish anything under the OGL.  

This would be like Microsoft saying that developers for Windows Vista are forbidden from publishing anything under the GNU public license.  And the upshot is now that developers have to choose between not developing games with the improved system or destroying their back-catalogs. 

Even if you don’t have a huge interest in D&D – in which case, I envy your normal social adjustment and relatively less awkward adolescence – it pays to keep up with this developing situation to see how a fight to close an open-source software product might actually go down.  Will Hasbro fail in its efforts to dominate the RPG industry, either shrinking their portion of market share or shrinking the size of the entire market?  Or will Hasbro succeed with this business plan, and the publishers of Monopoly (the game) end up with a de facto monopoly (the economic term) on this niche industry?

Update: On May 2, 2008, a week after this article's publication, Wizards of the Coast released an FAQ about the 4th edition licensing terms. The FAQ states:

Q. Can companies still produce 3.x products under the OGL?
A. Yes, but we anticipate that interest in the 4e GSLs will be greater.

Q. Can publishers release new products under both the OGL and 4E GSL?
A. No. Each new product will be either OGL or 4E GSL. If a new product is published under the 4e GSL, it cannot also be published as 3.x product under the OGL; and vice versa.

Q. I have multiple product lines. If I update one product line to 4th Edition, do they all have to be updated?
A. No. Publishers are able to choose on a product line by product line basis which license will work best.

Q. Will there be a different license for other lines, such as d20 Modern?
A. The d20 GSL will allow for other genres of roleplaying games.

Q. Why are there two different licenses?
A. The D&D 4e GSL is specific to the Dungeons & Dragons brand. The d20 GSL allows for non-fantasy genres. Both licenses tie to the 4th edition rule set.

Q. Do I have to give up my right to publish 3.5 OGL products in order to publish 4e compatible products?
A. No. Publishers are free to print product lines under either the OGL or 4E GSL. We would love to see our industry colleagues convert their entire product offerings to 4E, as we are doing, but we do not expect or require entire companies to convert to the new edition.

Q. Can publishers update their previous publications from older editions to the D&D 4th Edition rules?
A. Yes. Publishers participating in the Dungeons & Dragons 4th Edition GSL will be allowed, and encouraged, to convert their publications from earlier editions to the 4th Edition rules.

In a few minutes, Jim Metzler of Ashton, Metzler, and Associates, will be delivering his keynote on the Next Generation NOC at NetQoS Symposium 2008 at Barton Creek Resort in Austin. Last week, we pre-recorded a podcast with Dr. Metzler regarding the speech he is about to give and what he means by a "next generation NOC."

He talks about the changing role of the NOC and moves in enterprises towards integrating what were once seperate stovepipe functions to focus on application delivery.

The podcast is below.

CIA Plan #328 to remove Castro from power in Cuba has succeeded. This plan, also known as "wait until El Presidente gets old and retires," now suggests that there will be vast geopolitical changes.

However, while Castro may be retiring, the communist regime he's supported for nearly half a century isn't going away, and whoever wins the elections in Cuba, the policies vis-à-vis the U.S. probably won't change that much until and unless the U.S. foreign policy towards Cuba drastically changes after the next U.S. election.

Which is a shame, really, because when old hardware (I have no problem calling Castro "old hardware,") is too troublesome to maintain than to replace, it's usually a good time to re-evaluate the way that your company, or, in some cases, your tropical island nation, does business.

Now, whether Castro is retiring because of his natural health, or because the CIA slipped him a cigar with a really, really slow acting poison back in 1963, it doesn't matter. The point is that we are so used to the way things are, we often don't pause to consider what could be. We don't understand the true cost of maintaining legacy hardware versus trying the new, and don't give much thought to better alternatives if "things are working well enough."

IT professionals have been thwarted many times by using legacy management tools that are outmoded, increasingly and annoyingly cumbersome to maintain and administer, or no longer deliver the value for which they were acquired. Beyond the obvious maintenance and license costs, there are also the opportunity costs related to speed, quality, scalability and efficiency. When you combine high maintenance costs with opportunity costs, the financial penalty for the status quo begins to add up.

Now, no one should advocate tossing out a product or technology just because it's a little long in the tooth- in fact, long, huge, revolutionary solutions when small fixes are the most efficient solution can be a serious problem that takes time and energy away from IT. But, there is nothing wrong with taking time to assess the situation and ask whether we're better served by what is or what could be.

The Internet Corporation for Assigned Names and Numbers (ICANN) recently put out a press release which announced that six of the 13 root servers in the root zone (presumably located in-between the Phantom Zone and the Forbidden Zone) now had IPv6 addresses.

It's a small step but one which is necessary for bigger steps to follow. With the root name servers having IPv6 addresses, it paves the way for a full IPv6 end-to-end transmission path for data. The fact that the nameservers still relied on IPv4 made at least some form of IPv4 over IPv6 necessary for Internet transmission.

We were able to ask a few questions about the IPv6 assignment to ICANN and got answers from David Conrad, Vice President of Research and IANA Strategy.

NPD: Why only six of the 13 root servers?

Conrad: 6 of the 13 were ready at this time. Each of the root servers is run independently and are funded through internal means (that is, in general, no one is paying any of the root server operators to operate a root server directly). The 6 that requested ICANN add IPv6 records were the ones that had finished setting up their IPv6 infrastructure sufficiently to provide service.

NPD: Could you explain a bit about the 512 byte limit on the packet sizes?

Conrad: The original specification of the DNS protocol chose 512 bytes as a reasonable approximation of the largest packet that could get through the Internet (of the time, circa 1983) without being fragmented. Enhancements to the DNS protocol since then have allowed for an increase in that limit (specifically, requesters can indicate how large a packet they're willing to accept).

NPD: How will computers get the new info about the root name servers' new IP addresses?

Conrad: The only computers that will actually need the new information are DNS caching servers. When a caching server starts up, it asks one of the 13 root servers it has pre-configured (the root hints) for an up-to-date list of all the root servers. It then uses that new list.

DNS caching servers are typically operated by ISPs or the IT departments of large enterprises. Average PCs and workstations send their DNS queries to these caching servers.

NPD: Would this require operating system upgrades/patches?

Conrad: A patch will likely be supplied to make the change in the root hints permanent (the updated list obtained by the caching servers isn't generally written to disk), but as described previously, caching servers will be able to use the new addresses without the patch.

Is this a significant move towards standardizing Internet traffic on IPv6? Tell us your thoughts in our comments section below.

Recently, Network Performance Daily did a story on the Cisco Nexus 7000 switch, which had recently been announced by Cisco and will likely be a very important piece of enterprise hardware.

After our article, Douglas Gourlay, the Senior Director of Marketing and Product Management of Cisco's Data Center Business Unit, contacted us and pointed out that we were mistaken about some of the capabilities of the Cisco Nexus 7000 and so we invited him to do this podcast with us.

Douglas Gourlay from Cisco has posted a comment on this site which probably explains some of the features of the Nexus 7000 much better than I possibly could. Please check it out.

Cisco just announced its Nexus 7000 switch will be available in the second half of 2008.

The Nexus is a rather large machine that boasts a number of improvements over the 6500 series switches. If you have recently made a large purchase of 6500 series switches for your enterprise, you have my sympathy.

Andy Greenberg covers the technical beat at Forbes.com and explains some of the benefits of the behemoth box in plain English. For example, the Nexus has Gigabit Ethernet support (as well as support for future delivery of 40Gb/s and 100Gb/s,) so that you can run your whole data center with just Ethernet.

The killer feature, according to Greenberg, is that the Nexus 7000 allows distant virtual servers to use parallel processing. In other words, the virtual server in San Juan can now take advantage of idle cycles on the virtual server in Philadelphia, and vice versa. But there's more to it than that. The Cisco NX-OS, the new OS for the Nexus 7000, is said to allow switches to be virtualized at the device level, doing for switches what VMWare did for servers - that is, it not only helps get virtualized systems into parallel processing clusters but network engineers can create "domains" which define particular combinations of "bandwidth, processing, storage, and software," according to an article in BusinessWeek.

In addition to the obvious benefits, using previously unused resources can do a great amount towards reducing energy consumption and costs.

For a real look at the Nexus 7000, Cisco has provided an interactive model of the Cisco Nexus 7000 on their Web site.

What are your thoughts on the Cisco Nexus 7000? Feel free to discuss it in our comments section.

By Brian Boyko

Fingerpointing - it's a frustrating and lingering problem for IT organizations. Whenever, application performance degrades, all of a sudden application, server, and networking teams start pointing the finger at one another in an attempt to pass the blame. But isn't that why we have network monitoring tools in the first place - to tell you where the problem is, so that you can fix it faster.

Theoretically, yes. Unfortunately, network teams may be suffering from an undeserved "credibility gap" that prevents companies from taking timely action when problems arise.

For so very long, problems with network performance have often been laid at the feet of the networking team because, quite frankly, to the end user, an application problem, a server problem, and a network link problem all look like the same thing. "The network is slow." So even when it's not the network, the network team often gets the blame.

(Continued...)

Continue reading "Fingerpointing, Frustrated Network Engineers, and the Application Performance Blame Game" »

Ben Erwin, product manager at NetQoS, quickly explains the impact of WAN Optimization on TCP-based Applications in a short video, kicking off our "Whiteboard Series."

If you have questions about the video, please leave a comment below and we'll do our best to answer them.
--------------
More information:

On WAN Optimization:
- NetQoS and Cisco Webinar: Evaluating WAN Optimization Options and Quantifying the Results

Also in our Whiteboard Series:
- The impact of WAN Optimization on NetFlow/IPFIX measurements

With Ben Erwin:
- Third Party Integration and UI in the Enterprise