Previous incidents

Cache API recovered.

This wasn't a cache outage. Instead the cache was just not returning a response to the healthcheck in time and our PaaS provder timed out.

New caching layer fallback approach to be deployed on Monday where the requests will fallback to a direct API request should the cache go down.

UI recovered.

4 mins of downtime due to a massive traffic spike.

We’re looking into why our new caching layer didn’t prevent this.

We upped the client-ui resources and frontend sites recovered immediately, but we have tested this kind of spike before, so we’re looking into why this caused service interruption.

UI recovered.

1 minute downtime recorded on one client only. We believe this is due to a caching layer server restart not becoming available in time at the daily restart.

We're looking to implement a fallback option for where the caching layer is unresponsive which would also cover this particular fringe-case.