Network Performance Archives

Aberdeen Network Management Report Validates Our Strategic Approach


The Aberdeen Group, a provider of business research services surveyed 205 organizations last month to identify best practices for enterprise network visibility initiatives and controls. They called the report "The Real Value of Network Visibility."

In the interests of full disclosure, it should be said that NetQoS co-sponsored the study but we did so only after the survey was conducted and the analysis complete. That said, though, the study pretty much validates our entire "performance first" approach towards network and application performance management.

What the Aberdeen Group suggests is a PACE model (Pressures, Actions, Capabilities, and Enablers) to achieve corporate goals. The idea is that businesses are pressured to be responsible to customer needs, and the actions that are effective are to establish a proactive control of the network. In order to do this, you need to be capable of defining your escalation pathways for network performance issues, having normal networking performance baselines, understand interdependencies between applications on the network, be able to segment round-trip application response times into delays caused by the server, the network, and the application, and finally, have a centralized point for looking at the network performance data.

Frequent readers of this blog will no doubt notice that this is the point where I usually mention that NetQoS makes some of the products which enable those capabilities. The Aberdeen Group reports that these "enablers" are network performance monitoring through a Web interface, tools for remote analysis and troubleshooting of network performance, tools for creating custom profiles for monitoring groups of network hardware, a unified network performance and security platform, tools for Netflow data analysis, and a lab environment to simulate network performance.

There are some other gems in there to be found. The survey results showed that that top 20% of performance scorers:

  • Were the most likely to have the capabilities and enablers mentioned under the PACE model.
  • Were spending less time on troubleshooting network performance and application performance, managing changes to network design, or enforcing network usage policies.
  • Were much more likely to have merged application and network management into a single job role, and more likely to merge the application, network, and systems management teams into a single organizational unit.
  • Were able to fix problems faster and less likely to rely on calls to the help desk for determining network problems.

What do you think about the Aberdeen Group's report? Feel free to leave a comment below.


Network Performance Archives

Is Time Warner's "pay-as-you-go" trial good network management planning? Probably not.


brianboyko3.jpgEditorial by Brian Boyko
Editor, Network Performance Daily

Ars Technica reported that a memo claiming that Time Warner was going to roll out a "pay-as-you-go" metered scheme for Internet access, rather than today's subscription-based unlimited bandwidth access plans was leaked to BroadbandReports.com. That memo, which since has been removed, claimed that Time Warner was going to try metered/limited access on a trial basis in Beaumont, Texas, and Time Warner representatives have confirmed this plan with Reuters.

As Ars Technica pointed out, Comcast has tried using bandwidth caps and traffic shaping to curb Internet usage among the customers that pay Comcast for Internet access. Comcast, however, has run into trouble because it has not revealed those policies to Comcast's customers. Time Warner will supposedly give customers online tools to monitor bandwidth usage.

Of course, it would be the best solution to increase the capability of the network - ISPs have to play by different rules than corporate networks as they are common carriers. But we don't know whether it is economically feasible for Time Warner's cable division to remain profitable while increasing the bandwidth, and if an unlimited-access plan is not feasible, a pay-as-you-go plan seems at first to be the fairest of the alternatives.

That said, there's something a little, well, strange about this, because the Internet is not a big truck that you just dump something on. It's a series of tubes.

Solving The Wrong Problem

That is, all Internet connections are merely the transfer of little positively and negatively charged electrical bits which stream down the wire. The limitations are not in the availability of the resource but in the capacity of distribution. We are not, in other words, "running out of bandwidth" like we run out of oil, run out of water, or run out of diapers.

What is limited is the capacity of the "pipe." To strain a metaphor, you could push Lake Michigan through a coffee stirring straw, but it would take a very, very long time.

Any pay-as-you-go plan has a fatal flaw - it doesn't make a whole lot of sense to bill people for the data they are downloading because data is not the limited resource!

What is limited is the capacity of the ISP's infrastructure at any particular moment in time, so it would be saner to limit the usage of the pipeline at a particular time. Perhaps to even out the usage of bandwidth, the ISP could provide different speeds for peak and off-peak usage times. Those unhappy with the slow speeds at peak times could pay a premium for a greater share of the pipe.

But wait a minute! ISPs already do this - I know that my Internet connection at home is capped at a certain speed. In fact I could get a faster speed simply by asking for it and paying a premium - no delay nor needed infrastructure upgrades. Just cash.

So the move to a pay-as-you-go plan seems, to be at best a case of solving the wrong problem, and at worst a case of "double dipping" by making people pay for data and bandwidth. (If there are network slowdowns, charging people per-gigabyte won't help much if people are still downloading that gigabyte at the same time of the day, after all.)

Your Experiences May Differ

Unfortunately, I've been on the receiving ends of one of these plans. Recently I was in New Zealand filming a movie about electoral reform. Bland stuff. While I was there, I was planning to upload film to the Internet - sort of a production blog. But I found that I couldn't - the ISP there, New Zealand Telecom, had placed my flatmates and myself on a pay-as-you-go program with a cap of only one gigabyte, and they would not increase the cap until the next billing period, which would have been after I left the country.

One gigabyte. Anything over that amount was downloaded at speeds that I hadn't seen since I bought my last 56.6k modem. That meant that even doing things like normal Web browsing was a particularly hard chore. Uploading film to YouTube was right out. I was even hoping to get some extra work done for Network Performance Daily during that time but found that I simply did not have the ability to do so. I was, in a word, ticked off and frustrated. It certainly made it quite a bit harder for me to use the network - I ended up getting a lot of iced mochas at the local Internet café, as patronage was a prerequisite for Internet service.

Now, I have no idea if Time Warner plans anything like New Zealand Telecom, and Time Warner has more competition - even in Beaumont, TX - than New Zealand Telecom did in Wellington. That may force them to abandon this plan if they find customers cancelling accounts and leaving for competitors.

It is rather important to notice that the last mainstream successful service that charged you based on how much you used it was 1996's AOL.

I've never been to Japan, France, or Korea but I'm told that all of these countries have broadband available at much greater speed, without having to worry about pay-as-you-go plans. So the question is not whether unlimited broadband is technically feasible as more people use broadband, the question is whether companies are willing to make the infrastructure investments necessary. And considering that there will be more competition, not less, as new technologies (like FIOS and WiMax) become available, investing in infrastructure rather than limiting customers seems to be the smarter move in the long term.

But let's say that this plan is a success in Beaumont, and catches on. What's the upshot for enterprise networking?

You Think You Have A Recreational Network Use Problem Now…

If people come to expect that every piece of data that goes through their network is going to cost them extra money, that may mean that all the large data that they were once downloading at home now ends up getting downloaded to the corporate network and taken home via flash drives. In addition to the spike in traffic use, there are also issues with copyright infringement liability, computer security (with flash drives from home possibly containing malware - not to mention that people will probably swap flash drives within the company, spreading infections,) and people looking for large files to download before they go home instead of doing work.

Now, in many ways, the problem with limited bandwidth availability from an ISP may seem similar to limited bandwidth availability on a corporate WAN. But a business has many more options for dealing with slow networks than an ISP does. Businesses can check their application performance and if necessary recode them (many legacy apps designed for a LAN are too "chatty" for the WAN.) They can set QoS policies to make sure certain types of traffic from certain types of applications get priority. Traffic can be rescheduled so that it goes through the system during off-peak times.

Businesses have all these options - including limiting the end-users in a number of different ways - because in a business, the network is there to serve the business. But in an ISP, the network is there to serve the subscribers by providing a common-carrier communications service.

As such, the subscribers of an ISP can and should determine what traffic should be on the network, when, where, and how much. Any methods to alter, curb, slow, or block traffic from the network should be disclosed to the end-user at the very least and should be avoided unless there are no other alternatives - to do otherwise is to create a value judgment on certain types of traffic and to endorse certain types of speech over others.

(Perhaps I'm wrong on this, but…) To my knowledge, no company uses a method similar to "pay-as-you-go" to curb recreational traffic on their networks. They may limit speeds to certain applications, they may block sites, but I don't believe that any company institutes a bandwidth cap on its own employees.

That to me suggests that this plan doesn't have much merit as a solution to ISP oversubscription.

What do you think about Time Warner's plan? Disagree with the author? Feel free to make your opinions heard in our comments section.


Network Performance Archives

The IPv6 agenda: What government delays mean for network engineering plans.


brianboyko3.jpgBy Brian Boyko
Editor, Network Performance Daily

2008. Leap year. Election year. Government deadline for IPv6.

With the exception of the military, federal government agency CIOs are doing the bare minimum required by law to meet the mandate to move to the IPv6 network protocol by June, 2008. That bare minimum, according to this Network World story, is for all the hardware to be IPv6 capable - but does not actually require any IPv6 traffic to go across the network.

Whether or not this delay is a pragmatic consideration or the path of least resistance, most of you are probably having the same reaction regardless of your ideological bent. Liberals can call it typical of the current administration; conservatives can call it typical of government in general. Either way, the reaction that is probably shared by most of us consists of rolled eyes, a resigned sigh, and the word "typical" muttered slightly louder than a whisper under our breath.

For a contemporary example, the HDTV changeover, for example, has been looming "on the horizon" for decades now. At least one person I've talked to in the company has bought what they thought would be the "last SDTV they ever bought" four times since then. I'm still using an NTSC, instead of ATSC (high-def), tuner in my media PC because I don't believe that NTSC broadcasts will cease anytime soon, despite the February 2009 hard shut-off date.

Let's face it. It's hard to get the government to move on something until they have no other choice. This differs from the corporate world in so far as it's impossible to get a company to move on something until you can prove that they have no other choice. Which is one of the reasons government adoption of IPv6 is such an important issue.

Companies are not going to move to IPv6 unless it can be proven that the multiple and numerous drawbacks - reconfiguring an entire network, making sure existing applications (including network management software) are compatible, reconfiguring security, and, oh yeah, that looming $200B infrastructure cost - outweigh the benefits. And of course, backwards compatibility with IPv4 must be maintained until almost everyone is on IPv6.

Additionally, because of NAT and similar technologies (which have some beneficial side effects), the problem can often seem like it can be perpetually postponed. NAT solutions, of course, break P2P apps, impact network robustness by having an additional potential point of failure, and add complexity to the design and deployment of networks. Trying to solve IPv4 address exhaustion with more NAT is like trying to solve global warming with more air conditioning.

And the benefit is that you'll be able to add new devices to the network, and that configuration will be more in-line with the "end-to-end principle." There are some advantages, such as auto-configuration and mobility, some built-in QoS configuration, some new ways to deal with end-user computer security and better routing scalability, but many of these problems have existing solutions in IPv4. Those solutions may be more complex, harder to maintain, and may be less optimal for your network performance, but they exist. So, in short, IPv6's main selling point is that you'll be able to do the same things as you currently do on your IPv4 network.

If that's the carrot, what's the stick? Without it, you'll be less able to communicate with all the other networks that exclusively use IPv6. This does include China, which has already made the switchover, but does not, as stated earlier, include the U.S. government.

You can see why this can be a tough sell. Tons of work for IT, at a time when everyone's got some major project like a VoIP rollout whose benefits are a bit less ephemeral and a bit more quantifiable - all because we're running out of IP addresses.

We've faced problems like this before. But the prospect of running out of IP addresses isn't as sexy or scary as some of the nightmare scenarios that were associated with the Y2K problem. People were stockpiling cans of beans in basements waiting for the Russian nukes to fly and robots to revolt and for anything more sophisticated than the steam-powered abacus to stop working or something like that.

In response to the unadulterated panic associated with Y2K, the U.S. government passed the "Year 2000 Information and Readiness Disclosure Act," which worked with private companies to solve the problem. So what will it take to get governments and corporations to move in a timely manner? Apparently, threat of nuclear Armageddon and a ticking digital clock.

So, until we convince the writers of "24" that Jack Bauer's next adventure should deal with terrorists threatening to blow up unallocated IP addresses, the best plan to move to IPv6 is to get the government to move to IPv6, which will get the government contractors to move to IPv6, which will get everyone else to move to IPv6.

Without IPv6, there is a theory that an impending IPv4 address exhaustion will result in a speculative market on IPv4 addresses. That may very well occur, but such a market would only postpone absolute exhaustion of IPv4, trading of addresses would result in fragmented allocation patterns that would fragment and expand routing tables - older or low-powered routers may not be able to handle the increased load, and, since each change in an IP number requires massive changes to network configurations, there's little market liquidity in it. In short, the development of an IPv4 market might be so disruptive to the operations of the network that IPv6 might be less disruptive.

So, the fact that the government isn't actually moving to IPv6 despite the implied promise that it would, is, therefore, a problem.

Take away from this a simple lesson: sometimes it takes a few bold actors to move forward with broad change even if it seems early to do so. Without those bold actors, the inevitable change may take much longer to reach the tipping point - causing pain and strain for everyone.

What motivates you to changeover to IPv6, or what motivations would you need to change to IPv6? Feel free to leave a comment in our comments section.


Network Performance Archives

Network Management Challenges in Full-Throttle Russia: NetQoS Helps Moscow's Top Organizations with Network Visibility


russia1.jpg

By Nathan Bragaw
Business Development Manager, NetQoS

NetQoS European Tech Rep. Peter Frame and I made a trip to Moscow last month to attend a network management event hosted by our new Russian partner, UNICORNS. I thought I'd share a few observations from that trip.

First, the obvious: Russia is cold. I was born in Maine and been in Minnesota in Winter, but Russia, complete with snowfall and brisk wind qualifies as the coldest place I've ever been, and that includes Minnesota in Winter. On the plus side, the food is fantastic - I didn't realize I liked borscht until I ate at Taras Bulba on Petrovka Street - wonderful Ukranian and Georgian food.

It was clear from the wonderful work our local partner, Boris Goldshteyn, did with getting some of the biggest companies in Russia to attend our first network management event in the country that the Russian economy is in full throttle.

I am always fascinated by how different cultures and environments change the needs of our customers. In Russia, it's not that their network performance problems are any different. Instead, they are facing the same problems, like VoIP performance and WAN Optimization difficulties, but they are moving at a faster rate. This is largely because they are not confined by the fiscal restraints that companies in the U.S. face due to the fallout of the dot-bomb era. Companies in Russia want to invest in technology but are challenged to ensure that the technology they seek will meet the needs of their business. The network engineers we talked to there are looking at massive growth, the push for more current technology, and demands for understanding application performance over their networks.

russia2.jpg
These are the perfect customers for NetQoS. And I look forward to 2008 as we begin supplying solutions for Russian network engineers and network managers as they continue to get control over the massive changes that their networks are undergoing. We will be there with them helping them understand the impact of these changes and aiding them in proving that the network is effective for application delivery to their end users.

What differences in culture and environments have you found when working on multinational IT projects? Feel free to tell us about them in the comments section of this post.


Network Performance Archives

Get Your Network in Shape for 2008 With the Network Performance Daily Reader Contest


Submit Your Network Performance Management Resolutions for 2008 by January 3rd and Win a Nintendo Wii, TomTom or iPod

Over the past year, Network Performance Daily readers have submitted some of the best ideas and fodder for blog posts. We’ve written about everything from IT politics in Fingerpointing, Frustrated Network Engineers, and the Application Performance Blame Game to technical challenges in Echo In Voice over IP Systems, and the difficulties in changing behavior in God Help the Help Desk.

Continue reading "Get Your Network in Shape for 2008 With the Network Performance Daily Reader Contest" »


Network Performance Archives

Cisco IPSLA and NetFlow Help Network Managers Monitor VoIP Performance


We’ve been blogging with excerpts from our new VoIP Performance ebook here on Network Performance Daily. Today, Brad Reese gets into the game with his own advice on monitoring VoIP performance using Cisco NetFlow and IP SLA in a story on his Cisco Subnet blog called Are you Taking Advantage of NetFlow and IP SLA?

Brad quotes NetQoS CEO Joel Trammell on the subject. According to Joel, “the number one killer of voice traffic is network latency and jitter. Latency, jitter and packet loss cause poor audio quality and dropped calls. Latency caused by overloaded call managers or network congestion can be a major cause of poor VoIP performance."

Continue reading "Cisco IPSLA and NetFlow Help Network Managers Monitor VoIP Performance" »


Shaking Up Network Performance Training and the Annual NetQoS Symposium. Follow the Developments on Twitter.


NetQoS customers are in for a powerhouse educational event next year. The newly revamped NetQoS Customer Symposium is slated to be held April 20 – 23 at the Barton Creek Resort in Austin, Texas. We’ve completely redesigned the agenda to accommodate lots of new network performance best practices, tutorials, and hands-on lab time. We’re also signing up some big name speakers and counting on Dr. Jim Metzler, again this year, to dazzle us with thought-provoking new findings in the area of Application Delivery. If you’re interested, follow NetQoSLive on Twitter to track new developments and speakers as they sign on.

Based on lots of great customer feedback, we’ve expanded the conference from three to four tracks so that attendees have more options. And, we heard you loud and clear: 2008 will be all about network performance best practices. A “birds of a feather” session has also been added to the agenda so you can discuss the topics that interest you the most in a focused group. The NetFlow® training has been condensed to a four hour boot camp, allowing you to hone your traffic analysis skills in a shorter amount of time. As always, you will receive extensive training on NetQoS products with new topics on WAN Optimization, Packet-Level Forensics, VoIP Monitoring, and others.

Continue reading "Shaking Up Network Performance Training and the Annual NetQoS Symposium. Follow the Developments on Twitter." »


Network Performance Archives

P2P2B2B: Whatever happened to the promised P2P business apps?


brianboyko3.jpgBy Brian Boyko
Editor, Network Performance Daily

There are a couple of big stories regarding consumer P2P today. Trent Reznor, fed up with his label's pricing of his albums in Australia has told his fans directly that they should "steal it," at a Sydney concert. A video of the concert, leaked to YouTube, made it to the front pages of many social news sites.

I can't endorse Reznor's suggested course of action, but I'm not going to not endorse it.

At the same time, e-mails leaked from MediaDefender, a company that patrols P2P networks looking for copyright violators, showed that, despite MediaDefender's prior public assurances, they were operating a video uploading site called MiiVi, which many suspected was a "trap" to lure copyright infringers - among other concerns.

And with all this coverage of P2P technology in the news, I thought - what ever happened to those promised P2P corporate networking apps? Back in 2005, Network World mentioned "Groove Networks Virtual Office." In that story, the author states:

"With the adoption of Globus 4.0 as the new XML-based protocol standard, grid services will become the P2P of Web services…."

"Be forewarned, get educated and be prepared for the network implications of corporate IT P2P applications. The corporate next-generation network future may be just around the corner in 2006."

Searching for Groove Networks on Google leads to Microsoft's Web site - It's an add-on for Microsoft Office now. Not surprising, as Groove's founder, Ray Ozzie became Chief Software Archetict at Microsoft. (He was also the brains behind Lotus Notes)

I looked up Globus as well. Globus's is more of a project to create a computing grid of corporate network computers - which is a pretty cool idea. 90% of the time, I'm not using 100% of my dual-core processor computer here at work. The other 10% of the time, I really could use more processing power to render video. If there's a way to grid-together peer-to-peer corporate computers so that everyone gets the power they need to do their jobs, sharing CPU power over the network.

Still, I don't think most companies are ready to go that far - mostly, I think, they just want their files faster and their latency lower.

So, where are the corporate P2P applications? Maybe I'm overlooking them (and if so, I'd love for contributors to point out projects to me in the comments) but it seems like this is an area of technology that was abandoned too quickly, and maybe deserves reexamination.

----------------------------

(N.B. We tried to contact Ray Ozzie at Microsoft for his thoughts on this matter, but was told by a representative of Waggener/Edstrom that Microsoft was unable to participate in this particular opportunity at this time.)



Network Performance Archives

Software as a disservice: Why you can't always rely on SaaS


brianboyko3.jpgBy Brian Boyko
Editor, Network Performance Daily

We were supposed to have the video of Dr. Steve Fulton we put up on Thursday night up by Wednesday afternoon. That didn't happen.

By coincidence, we also saw this opinion piece by John Dvorak, and linked to it the day before we needed the video to go up. The article, entitled "Don't trust the servers," talks about problems with Windows Genuine Advantage and how it illustrates the problems with SaaS solutions - that you're eternally dependent on a third party to continue to provide service.

In our case, we rely on Google Video to provide the bandwidth and hosting for the videos in our Whiteboard series. Now, you can say what you want about "getting what you pay for" and the like, but when we couldn't upload our video to Google, I realized how dependent I had gotten on their SaaS video hosting solution.

I couldn't log-in to upload the videos. At first I thought the problem was at my end - perhaps Symantec Anti-Virus had caused some sort of conflict or had firewalled off the ports that Google's video uploader needed. That wasn't it. Maybe it was something with our in-house network. That wasn't it either, as I found out when I took a copy of the video home and tried to upload it from both my Windows XP and my Linux partition. Nada. Zip. Zilch.

Despite the fact that Google's help files had a ready - and wrong - answer to every problem I came across, the problem was entirely Google's. And there was nothing I could do about it until they finally fixed whatever the problem was the next day.

See, while there are a number of video hosting solutions out there, Google Video and Veoh were the only ones that allowed me to upload clips more than 10 minutes long - and Veoh's playback was poor. Even if I could re-code the entire video at a lower resolution to lower the filesize, that didn't matter. Ten minutes was the hard limit on YouTube (also owned by Google) and other sites. My 25 minute video needed Google Video.

Eventually I was able to get the video uploaded, and though it took a while to process, it went up last night.

This isn't the first problem we've had with SaaS. Expensable.com often goes down for a few hours and we can't log expense reports. I use Gmail for my personal e-mail and while it's generally reliable, it does have some problems.

All in all, if you're looking at it from a productivity or a network performance view, moving your apps from the local network to a third party service - well, yes, it will absolutely save bandwidth and may make the network run faster for your other apps. But having a faster network doesn't mean anything if the end-user is waiting for a third-party service as long or longer as they used to wait for their slow-loading WAN apps. Or, in other words, you're not solving the problem of slow performance from the perspective of the end-user. You're just shifting blame.

This is not to rag on SaaS. I haven't lost an e-mail since I started using Gmail in 2004. My Flickr account saved pictures of my deceased friend John when my hard drive stopped working one day. And if it wasn't for YouTube, I'd never be able to show my parents in Virginia anything about my life in Texas.

But these experiences are a bit of a wake-up call that SaaS isn't going to solve every problem. Ultimately, the end goal of enterprise network performance is not to keep the network running as fast as possible, or to free up as much bandwidth. It's about finding the solutions which make the end-users more productive and enabling the company to do more as a result.



Network Performance Archives

How to lose friends and make enemies: The Comcast Capacity Planning lesson


brianboyko3.jpgBy Brian Boyko
Editor, Network Performance Daily

Right now there's a bit of a brouhaha about Comcast high speed service. Many Comcast customers are finding themselves cut off from the service because of excessive usage.

To be fair, I was unable to find any reference where Comcast says that their broadband package is "unlimited." However, they fail to disclose what, exactly, "excessive usage" consists of in their Acceptable Use Policy.

I don't have a problem with Comcast limiting bandwidth. There's only so much traffic that their servers can handle, so much that can go down their pipes. Theoretically, limiting the use of the heaviest users would enable better service for the vast majority of users for whom speed is more important than volume.

(Of course, the cynical assume that Comcast is dropping high-usage customers because they're the least profitable and that supporting those users would require investing more in bandwidth and infrastructure - but we'll leave that theory alone for now.)

What I'm concerned about is people suddenly being disconnected from the Internet after passing a line that they know nothing about. I'm sympathetic - my Internet access was cut off without warning back in 1998 at The College of New Jersey, and that cost me a pretty well-paying part-time job as a Web designer. (There were other reasons, but this was a significant reason that I decided to transfer to New Jersey Institute of Technology the next semester.) If my home internet access was cut off today, I'd be at a serious disadvantage with my job editing this blog!

But it also worries me because I can't imagine this happening to a corporate customer. If an IT department asked "how much bandwidth do we have," that information would never be withheld from them. You can't do any meaningful capacity planning if how much capacity you have is kept hidden from you.

Disclosure is obviously the most important step, but there are other options that Comcast could take. Instead of cutting users off, it could throttle down speeds once a customer produces a set amount of traffic - The customer still has access to the Internet, but it doesn't take up quite so much bandwidth. While downloading Linux ISOs via Torrent are going to take longer, viewing YouTube and talking on Skype shouldn't be affected by reasonable, but lower, bandwidth caps.

At any rate, if Comcast simply couldn't keep up with the demand, then perhaps they need to consider billing as a pay-as-you-go service. Sure, we did away with hourly billing around when AOL switched to flat-rate service in 1996… but certainly, paying for the service that you use is probably very appealing to the vast majority of people paying $50 a month to do nothing more than check e-mail and Web browse.

Then again, there are other solutions which are probably preferable. Namely - improving the performance of Comcast's existing infrastructure, or adding capacity to Comcast's existing infrastructure. Apparently, though, both those solutions are more expensive than suddenly dropping a few customers from the rolls and engendering ill-will.



1 2 3 4 5 6 7 8