Tips for responding to alerts when you are on support.
In case of security incident, for example when alerted of a brute-force attack by Prometheus monitors, it may help to look at UAA logs.
The access logs is on the UAA VMs in /var/vcap/sys/log/uaa/localhost_access.log.
All other logs are sent to logit. You must Log into logit via Google[external link] before you can access kibana.
The CDN broker has a healthcheck endpoint which confirms whether all of its dependencies are working. If any are not working then the healthcheck endpoint returns a 500 response code and response body containing the errors.
Unfortunately, many types of failure event are not resolvable without getting in touch with AWS. In some circumstances you may be able to restore an instance using our point-in-time-restore instructions.
When failure monitor triggers, before starting any further investigation and work, we would like to notify our users of odd behaviour and let them know we’re working to solve this issue. Please see the How to obtain organisation managers to learn who to contact.
For more information on the 8 types of failure event, see the Amazon RDS Event Categories and Event Messages.
Our tenants are likely to run low on storage on their RDS instances over time. This is no reason to panic, but we would like to notify our users when this happens.
Ideally, you should retrieve a GUID of an instance from the tags associated with the metrics on the alert. Having that will help you establish the users to contact, as they may wish to scale up their instance in response. Please see the How to obtain organisation managers to learn who to contact.
We’ve prepared a script to obtain organisation, space, instance name and the list of managers connected to the instance you’re after.
Last time we received an alert for API latency, it was a UAA bug that caused instances to leak memory. It also resulted in high CPU usage and the quickest solution to restoring the platform was to restart both instances one after the other.
There may be different reasons for the latency to drop down on any of the VMs. We found out the issue by logging into the VM itself and dig through the logs.
In the case of the UAA latency incident, it turned out that the CF Upgrade we did previously, contained a UAA release introducing the memory leak. The solution was to upgrade UAA release on its own.
Read more in the UAA Downtime investigation story.
Users have previously experienced intermittent request timeouts and errors when accessing their apps. During a particular incident, we believe that requests timed out due to ELB node failures.
Details of the monitoring approach can be found in the README for our monitoring application.
We have been advised by AWS that if we believe we are experiencing an issue with one or more ELB nodes that we should raise a “high priority” support ticket with them.
Read more in the Incident Report.
cdn.tls.certificates.validity - number of days before expiry of certificates for CloudFront aliases (tenants’ custom domain names). This is for all CDNs, not just those created by the broker.
If the endpoints are misconfigured or the certificate is considered invalid for some other reason the value will fall to 0 and alert as expired/invalid.
If the certificates are due to expire, check AWS Certificate Manager for the validation settings, as they should validate automatically via DNS. For CDN certificates start with the CDN Broker’s logs to investigate why they failed to renew via Let’s Encrypt.
If you are seeing NO_DATA errors for this monitor then there may be a more fundimental connectivity issue to the reported endpoint.
The cf logs output of the paas-metrics app may contain additional information. This is deployed to the 'monitoring’ space of the 'admin’ org.
We generate several certificates for Cloud Foundry which need to be rotated regularly (most of them have a 1 year expiry).
We have a Concourse job which checks if we have certificates expiring in less than 30 days.
If we need to rotate any of the certificates please follow the intructions here.
If you see a warning from Trusted Advisor it should be addressed.
IAM: Access keys should be rotated every 90 days, you should chase the owner of the offending key, delete the key or you can use this script to help with rotating keys.
S3 Bucket Permission’s for gds-paas-*-cdn-broker-challenge are required to be publicly accesible so it is safe to mark these as excluded.
S3 Bucket Permission’s for gds-paas-*-ci-releases are required to be publicly accesible so it is safe to mark these as excluded.
S3 Bucket Permission’s for gds-paas-*-ci-releases-blobs are required to be publicly accesible so it is safe to mark these as excluded.
Security Group - Specified Ports Unrestricted for *-sshproxy: We intentionally allow port 2222 for cf ssh access so it is safe to mark this as excluded.
As this is the first monitor of this type please investigate the gorouters to discover the issue they are encountering. We have previously seen high resource usage (CPU and Memory) these should be checked in the first case.