You could say that San Franicsco had a network security issue, but that’s really not accurate. San Francisco’s network was secure. It’s just that it was secured against the wrong people.

Though the San Francisco story of Terry Childs is nearing a conclusion, it’s important to look back at this and see what lessons we can learn from it. Quite frankly, being able to learn something from it seems the only upside to this debacle.

If you’re not familiar with the story, Terry Childs was a network administrator for the City of San Francisco. He built the network from the ground up, maintained it, and was the sole rights manager for the network. But Childs supposedly felt so protective of his network that he allegedly decided not to give out those root passwords, configuration changes, and documentation after he was dismissed to colleagues that his lawyer claims: “had in the past maliciously damaged the system themselves, hindered his ability to maintain it, and shown complete indifference to maintaining it themselves.”

In other words, the network was Childs’ baby, and he wasn’t about to give it up to people he thought incompetent.

According to ComputerWorld, this situation is not rare in IT:

"Unfortunately, it is not that uncommon to come into a situation where one or two people have created a situation where not only are they the only ones that know what is going on, but they are the only ones that can do anything," said Lou Michael, director of network and infrastructure services in Virginia's Arlington County department of technology services.

The issue isn't just control over passwords, but also over documentation relating to configurations and changes. Often in situations such as this, "requests for access, passwords and documentation are frequently taken as hostile acts by those that have been holding the keys to the kingdom," he added. "In my experience, I have encountered this type of situation on more than one occasion," he said. In one incident, a mainframe systems programmer had to be fired for changing access rights because he disapproved of others' activities on the system, Michael said. In another case, the individual resigned when he "realized that the pressure to follow processes and procedures was not going to go away despite the protesting," Michael said.

In IT we do have a tendency to form attachments to the tools of our trade. It can be difficult, as network engineers know themselves to be artists and the maintainers of a complex system, one in which balance and perfection are key.

The grand irony is that much of what IT does is invisible to most people in the company, and the rest of what IT does is incomprehensible to most people in the company. Even if they see the grand opus, they don’t recognize it for the masterpiece that it is. Pigeons crapping on sculpture.

We’ve mentioned before the mistrust that occurs between network and application engineers and management, and how it can be a hard sell to give management even high level visibility into network performance. But network engineers need to also realize their role of supporting the business and help break down the silos between them and everyone else.

Otherwise, you end up with situations like the one that happened in San Francisco.