How can you set up alerts for critical system events?

Implementing proactive monitoring and incident response involves setting up alerts for critical system events. This involves identifying events, defining thresholds, choosing monitoring tools, logging, event correlation, automated scripts, and more. By implementing these practices, organizations can improve their ability to detect, respond to, and mitigate critical system events, ensuring the overall health and security of their IT infrastructure.

Most important in defining alert criteria is the identification of multiple smaller threshold events which if considered together indicate severe threats. So while the triggers for individual events may be below threshold but combined impact of smaller events together is severe. To conclude I will say that setting up trigger threshold is the single most important deciding parameter and should be done after detailed study and due diligence

É necessário definir primeiramente os crítérios para monitoração e tipos de alertas e o que deve ser monitorado. Somente a partir destas definições deve-se pensar em qual a melhor ferramenta e que tipo de estratégias serão utilizadas.

Warning. The criterion for defining alarms is ALWAYS risk analysis. The alarm is the second type of risk control that can be done, the first is intrinsically safe design, to achieve the elimination or reduction of risk. So if it is not possible to reduce or eliminate the risk, alarms are introduced. Even in identifying the threshold and the means of conveying the alarm (sound, visual, ..) it is important to rely on a risk analysis standard (the effectiveness of an alarm must be verified).

There are lots of different types of events that you may want to be alerted about. These could include: System errors: Application crashes, service interruptions, database failures. Performance issues: High CPU usage, memory leaks, slow response times. Security events: Unauthorized access attempts, suspicious activity, malware detection. Resource constraints: Low disk space, high network traffic, approaching capacity limits. You may have various tools to monitor and measure each for you. The device you pick, integrates with monitoring tools

Com as definições estratégicas em mãos, idealmente deve-se avaliar qual ferramenta está mais aderente ao tipo de monitoração que a empresa deseja implementar. Neste caso será necessário uma boa pesquisa nas funcionalidades de cada uma delas.

Tools for reporting alarms must always be verified with standards (e.g. ISO 80002-2). Using tools that have not been validated is very dangerous: remember that the tool that works well for others does not necessarily work well for you (the context changes the risks)

Remember: too many alarms = no alarms. Alarms must be few and extremely reliable. NEVER send unreliable alarms because the user will stop looking at them.

É essencial testar os alertas e revisar as configurações, evitando falsos positivos, chamados desnecessários ou alertas atrasados.

All mitigation measures must be verified. It is important to check them even in error conditions and those deemed impossible. The impossible happens!

Once you have a lot of alerts set up, it may help to have something that can help triage or route them to the right team member. I like PagerDuty for this. But OpsGenie and VictorOps also work quite well and serve the same function.

This is a critical step in the alerting process. I have seen the approach to alerting in some organisations to be set and forget, this is wrong, and will lead to alert fatigue or wasted resources. To ensure alerts act as valuable indicators rather than noise, ongoing management is crucial. This involves reviewing criteria, analyzing trends, and adapting to changing environments. Effective alerting is an ongoing journey.

The management of the alarms must be carried out under quality procedures: without certified procedures there is no security in the management and accuracy in the calibration of the thresholds. A mitigation that is not tested periodically is practically useless.

The management of the alarms must be carried out under quality procedures: without certified procedures there is no security in the management and accuracy in the calibration of the thresholds. A mitigation that is not tested periodically is practically useless.

One important aspect I wish to add here is the use of artificial intelligence something which we use in SOAR now because it's quite possible that some triggers get missed or not acted upon because of sheer amount of triggers. Hence use AI as far as possible to reduce the errors due to the human fatigue. To conclude I will say that we need to leverage the AI ML aspects to manager the triggers effectively and efficiently