Prometheus Alerting Rules That Don't Cry Wolf

The fastest way to get a team to ignore Prometheus is to point an alert at up == 0 and ship it. Week one it catches a real outage and everyone's impressed. By week three it has fired a couple dozen ti...