I frequently have the chance to present in front of real customers who come in to find out the latest that HP is doing during visits to the Executive Briefing Center here in Houston and also in Cupertino.  This morning was one of those times, but a little bit different, because the customer was a little smaller than those who usually visit (not that there's anything wrong with that).  It was the CIO and one of the members of the server admin team, and the company runs around 40 ProLiant servers from the DL300 series, and a StorageWorks SAN.

As I asked questions to get a better picture of how they managed things in the environment, it became apparent that their current state was something they arrived at accidentally; it wasn't planned.  Their practice was to load the ProLiant server management agents on the servers as they built them, but they didn't have any coordinated approach to how they used them.  The server admin in the meeting today said he thought one of his colleagues had "Insight Manager" set up some where, but he wasn't sure what version it was, how often it was used, and only that person had access to it.  In fact, the CIO had asked the server admin in the last week if he could report on how much memory was in their Citrix server, and he logged in to the ProLiant server management agents' System Management Homepage for each of them (luckily there were only 4) and was able to tell the CIO the information.

I thought that was a good example of an issue I've discussed with a lot of CIOs.  In most IT organizations, there are certain fixed costs, like facilities.  It takes some amount of power to keep the lights on and the cooling going, physical security, and so forth.  Not a lot of variability in those costs.  Likewise the hardware and software costs--sure you negotiate the best you can with your suppliers, but the biggest lever is always going to be volume, so unless you plan on doubling the number of servers you have in the next year, there isn't much to talk about.

What it gets down to is that for most IT organizations the cost that can be controlled the most is people cost.  Getting back to the customer this morning, that almost made the CIO fall out of his chair laughing, saying I'd made his day.  According to him, any control he had over his staff was at most an illusion.  So I asked a few more questions and got mostly the answers I expected, because I've seen it time after time.  His staff is made up of very skilled and experienced administrators.  The kind of dream team that you can throw anything at and expect them to deal with it.  But since you don't need the services of a "rocket scientist" every day, sometimes the tasks they do aren't always a match for their expertise.

I've said this many times to audiences around the world... (with some apologies for Jeff Foxworthy and his 'You might just be a redneck...' routine)

• If you see a Microsoft Certified System Engineer in your server room with a screwdriver in hand racking servers, you have a problem...
• If you see a Cisco Certified Network Administrator having to stand in front of a rack of equipment, you have a problem...
• If you see a Red Hat Certified Engineer with CDs in hand installing an application or operating system module, you have a problem...

I'm sure a lot of you just thought of an incident that happened recently in your own organization similar to that described above.  I know every time I talk about this with groups of customers I always see heads nodding in agreement.  In a way, we've all done this to ourselves.  By not getting ahead of the curve and really planning out how incidents are dealt with and implemented solid procedures that are replicatable and repeatable, we end up in a situation where we need an extremely experienced crew because Murphy's Law always applies.

Think about it.  If everyone who came into an ER, whether with a head laceration, broken bone or in full cardiac arrest all required immediate and continuous care from an experienced physician, it would be unmanageable (not to mention extremely costly).  When you come to an ER you get triaged, and depending on the severity of your complaint, may end up being cared for by an intern, resident, physician's assistant, or nurse for part or all of your visit.  How is this possible?  Because of well-defined procedures and escalation processes.

From a process and procedure perspective, HP Systems Insight Manager can be a big help.  If you utilize it properly, it can help to parcel out easy tasks like replacing a hot-plug hard drive to less experienced staff members.  You also can use it to document how situations have been solved, thereby building organizational knowledge.  While you may have enjoyed my story above and generally agreed with my logic and conclusions, we have the data to prove the benefits of using HP SIM from real customer reports.  What they report is that by using HP SIM, most administrators can double the number of servers they supervise.  Additionally, unplanned downtime can do down by as much as 77%, because of features like the pre-failure alerts on CPU, disk and memory, and the quality of information provided that can minimize downtime when it happens.  This is documented in a report published by IDC called "Gaining Business Value and ROI with HP Systems Insight Manager."  The methodology and quotes from the customers interviewed is available in the report.

But what do you think?  Reports are well and good, but do you have an experience to share on this topic as well?  Use the comments feature to talk back!