System Admin Maxim #4: Monitor your systems

 As a system admin you are responsible for the infrastructure that the rest of the organization depends on for their jobs.  To assure that this infrastructure does not break and inopportune times, you need to monitor it.  This means that you set up systems, scripts, processes, etc. so when something starts to get out of whack; the infrastructure will alert you that there is something going wrong and perhaps you may want to do something about it.

Sounds simple, but monitoring is somewhat of an art form because:

• The infrastructure you are monitoring is dynamic and constantly changing.  So you need to constantly be updating your monitoring system to reflect this.
• You need to decide what is important to monitor and when the system should alert you and how it should alert you.  For critical things and email will not cut it.  You want a text or push notification to your phone.  
• Is the alert a 'real' one or a false positive. If you get too many false positives you will start ignoring them to your own peril.
• What happens if the system it uses to alert you fails?  How do you know the monitoring system is really working?  These are questions you need to resolve - perhaps by having it send out an 'OK message every day or so.
• Certain failures (e.g. a switch or router) and make it look like large portions of your infrastructure failed when they did not so you get a zillion alerts when really you only needed one alert from the failed switch or router.
• What about partial failures where people notice slowdowns, but the system is running? What about gradual failures where a disk is slowly filling up until the system completely breaks? You need to figure out how to deal with these sorts of failures.
• What is 'normal'.  Most systems will produce lots of monitoring data, but until you work with the system for at least a year, you really do not understand what is 'normal' data and what is 'I am about the break badly' data in many cases.  I say at least a year, because most organizations have yearly, monthly, quarterly, and other cycles that may affect the system.
• What dose it mean to be outside of normal?  Some percentage like 10%.  Perhaps 1 or 2 standard deviations. Perhaps you can compare it to previous years data AND the system had not changed too much so you can say this is really different, perhaps I should check into it.

As rule of thumb, I have found that you need to keep the monitoring systems as simple as possible, because if they are not they will not be maintained and just rot away.  Determine first which sort of events are causing your infrastructure problems, then start out small monitoring things that lead up to those events.  Then gradually add in more things while not increasing the false positives or workload on yourself.