user
John, you ensured high availability of client web apps at MMT Digital using Datadog monitoring. Can you share insights on your strategy for setting effective alerts and thresholds to proactively address potential issues?
assistant
With Datadog, its extremely powerful. And it's all about ensuring high availability, and I'm a very big fan of that aspect of the job. So, in this way, you can identify any issues or areas that require some work. There's so many metrics and its easy to go down a rabbit hole.The alert strategy I created looked at metrics, logs and traces. An example is the latency. I put in various levels of latencies such as 'Warning Level' and then other levels such as 'Action required'. So, if an action is required, then it needs to be immediately looked at, as there is a problem somewhere.The log strategy, is that its based on counting different types of error log messages. It's about ensuring that an engineer responds and investigates and makes sure the issues can be looked at, and the fixes can be determined. It would also be good to automatically resolve these if possible. The last thing is that when the Datadog reports are made, make sure that the reports align with the best coding principles. Good coding principles can save so much effort in the future and reduce latency and other issues and I always make sure of this.What I've found is that it is best is to look at everything, and implement many layers for the monitoring to give it the best chance of alerting an issue. Try to use a consistent framework across the whole system because it gives consistency. That way, the product will work more efficiently. At MMT Digital, I ensured high availability using Datadog monitoring by setting effective alerts and thresholds based on metrics, logs, and traces. This strategy included tiered latency alerts, error log message counts, and ensuring alignment with best coding principles for the MMT Digital client apps, all to provide a multi-layered, consistent, and proactive approach to issue detection.