REL 5: How do you design interactions in a distributed system to mitigate or withstand failures?

Distributed systems rely on communications networks to interconnect components (such as servers or services). Your workload must operate reliably despite data loss or latency over these networks. Components of the distributed system must operate in a way that does not negatively impact other components or the workload. These best practices enable workloads to withstand stresses or failures, more quickly recover from them, and mitigate the impact of such impairments. The result is improved mean time to recovery (MTTR).

Resources

Retry, backoff, and jitter: AWS re:Invent 2019: Introducing The Amazon Builders’ Library (DOP328)
Error Retries and Exponential Backoff in AWS
Amazon API Gateway: Throttle API Requests for Better Throughput
The Amazon Builders' Library: Timeouts, retries, and backoff with jitter
The Amazon Builders' Library: Avoiding fallback in distributed systems
The Amazon Builders' Library: Avoiding insurmountable queue backlogs
The Amazon Builders' Library: Caching challenges and strategies
Well-Architected lab: Level 300: Implementing Health Checks and Managing Dependencies to Improve Reliability
CircuitBreaker (summarizes Circuit Breaker from “Release It!” book)
Michael Nygard “Release It! Design and Deploy Production-Ready Software”

Best Practices:

Improvement Plan

Implement graceful degradation to transform applicable hard dependencies into soft dependencies

  • Implement graceful degradation to transform applicable hard dependencies into soft dependencies: When a component's dependencies are unhealthy, the component itself can still function, although in a degraded manner. For example, when a dependency call fails, failover to a predetermined static response.
  • Throttle requests

  • Throttle requests: This is a mitigation pattern to respond to an unexpected increase in demand. Some requests are honored but those over a defined limit are rejected and return a message indicating they have been throttled. The expectation on clients is that they will back off and abandon the request or try again at a slower rate.
  • Control and limit retry calls

  • Control and limit retry calls: Use exponential backoff to retry after progressively longer intervals. Introduce jitter to randomize those retry intervals, and limit the maximum number of retries.
    Error Retries and Exponential Backoff in AWS
  • Fail fast and limit queues

  • Fail fast and limit queues: If the workload is unable to respond successfully to a request, then fail fast. This allows the releasing of resources associated with a request, and permits the service to recover if it’s running out of resources. If the workload is able to respond successfully but the rate of requests is too high, then use a queue to buffer requests instead. However, do not allow long queues that can result in serving stale requests that the client has already given up on.
  • Set client timeouts

  • Set both a connection timeout and a request timeout on any remote call, and generally on any call across processes: Many frameworks offer built-in timeout capabilities, but be careful as many have default values that are infinite or too high. A value that is too high reduces the usefulness of the timeout because resources continue to be consumed while the client waits for the timeout to occur. A too low value can generate increased traffic on the backend and increased latency because too many requests are retried. In some cases, this can lead to complete outages because all requests are being retried.
    AWS SDK: Retries and Timeouts
  • Make services stateless where possible

  • Make your applications stateless: Stateless applications enable horizontal scaling and are tolerant to the failure of an individual node.
  • Implement emergency levers

  • Implement emergency levers: These are rapid processes that may mitigate availability impact on your workload. They can be operated in the absence of a root cause. An ideal emergency lever reduces the cognitive burden on the resolvers to zero by providing fully deterministic activation and deactivation criteria. Example levers include blocking all robot traffic or serving a static response. Levers are often manual, but they can also be automated