REL 1: How do you manage service quotas and constraints?
For cloud-based workload architectures, there are service quotas (which are also referred to as service limits). These quotas exist to prevent accidentally provisioning more resources than you need and to limit request rates on API operations so as to protect services from abuse. There are also resource constraints, for example, the rate that you can push bits down a fiber-optic cable, or the amount of storage on a physical disk.
Resources
AWS Live re:Inforce 2019 - Service Quotas
What Is Service Quotas?
AWS Service Quotas (formerly referred to as service limits)
Amazon EC2 Service Limits
AWS Trusted Advisor Best Practice Checks (see the Service Limits section)
AWS
limit monitor on AWS answers
AWS Marketplace: CMDB products that help track limits
APN Partner: partners that can help with configuration management
Best Practices:
-
Aware of service quotas and constraints: You are aware of your default quotas and quota increase requests for your workload architecture. You additionally know which resource constraints, such as disk or network, are potentially impactful.
-
Manage service quotas across accounts and regions: If you are using multiple AWS accounts or AWS Regions, ensure that you request the appropriate quotas in all environments in which your production workloads run.
-
Accommodate fixed service quotas and constraints through architecture: Be aware of unchangeable service quotas and physical resources, and architect to prevent these from impacting reliability.
-
Monitor and manage quotas: Evaluate your potential usage and increase your quotas appropriately allowing for planned growth in usage.
-
Automate quota management: Implement tools to alert you when thresholds are being approached. By using AWS Service Quotas APIs, you can automate quota increase requests.
-
Ensure that a sufficient gap exists between the current quotas and the maximum usage to accommodate failover: When a resource fails, it may still be counted against quotas until its successfully terminated. Ensure that your quotas cover the overlap of all failed resources with replacements before the failed resources are terminated. You should consider an Availability Zone failure when calculating this gap.
Improvement Plan
Aware of service quotas and constraints
What is Service Quotas?
- Select relevant accounts and Regions based on your service requirements, latency, regulatory, and disaster recovery (DR) requirements
- Review AWS service quotas in the published documentation and Service Quotas
AWS Service Quotas (formerly referred to as limits)- Determine all the services your workload requires by looking at the deployment code
- Use AWS Config to find all AWS resources used in your AWS accounts
AWS Config Supported AWS Resource Types and Resource Relationships - You can also use your AWS CloudFormation to determine your AWS resources used. Look at the resources that were created either
in the AWS console or via the list-stack-resources CLI command. You can also see resources
configured to be deployed in the template itself.
Viewing AWS CloudFormation Stack Data and Resources on the AWS Management Console
AWS CLI for CloudFormation: list-stack-resources - Determine the service quotas that apply. Use the programmatically accessible information via Trusted Advisor and Service Quotas.
Manage service quotas across accounts and regions
What is Service Quotas?
- Select relevant accounts and Regions based on your service requirements, latency, regulatory, and disaster recovery (DR) requirements
- Review AWS service quotas in the published documentation and Service Quotas
AWS Service Quotas (formerly referred to as limits)- Determine all the services your workload requires by looking at the deployment code
- Use AWS Config to find all AWS resources used in your AWS accounts
AWS Config Supported AWS Resource Types and Resource Relationships - You can also use your AWS CloudFormation to determine your AWS resources used. Look at the resources that were created either
in the AWS console or via the list-stack-resources CLI command. You can also see resources
configured to be deployed in the template itself.
Viewing AWS CloudFormation Stack Data and Resources on the AWS Management Console
AWS CLI for CloudFormation: list-stack-resources - Determine the service quotas that apply. Use the programmatically accessible information via Trusted Advisor and Service Quotas.
Accommodate fixed service quotas and constraints through architecture
AWS Service Quotas
Monitor and manage quotas
- Capture current resource consumption (for example, buckets, instances, etc.): Use service API operations, such as the Amazon EC2 DescribeInstances API, to collect current resource consumption.
- Capture your current quotas: Use AWS Service Quotas, AWS Trusted Advisor, and AWS documentation
- Use AWS Service Quotas, an AWS service that helps you manage your quotas for over 100 AWS services from one location
- Use Trusted Advisor service limits to determine your current service limits
- Use service API operations to determine current service quotas where supported
- Keep a record of quota increases that have been requested, and their status: After a quota increase has been approved, ensure that you update your records to reflect the change to the quota.
Automate quota management
- Use Service Quotas and augment the service with an automated quota monitoring solution,
such as AWS Limit Monitor or an offering from AWS Marketplace
What is Service Quotas?
AWS limit monitor - Set up triggered responses based on quota thresholds, using Amazon SNS and AWS Service Quotas APIs
- Configure limit thresholds
- Integrate with change events from AWS Config, deployment pipelines, Amazon EventBridge, or third parties
- Set up triggers to take appropriate action on notifications and contact AWS Support when necessary
- Test automation
Ensure that a sufficient gap exists between the current quotas and the maximum usage
to accommodate failover
- Determine your service quotas, accounting for your deployment
patterns, availability requirements, and consumption growth
- Determine your reliability requirements (also known as your "number of 9's")
- Establish your fault scenarios (for example, loss of a component, an Availability Zone, or a Region)
- Establish your deployment methodology (for example, Canary, Blue/Green, Red/Black, or rolling)
- Include an appropriate buffer (for example, 15%) to the current limit
- Plan consumption growth (for example, monitor your trends in consumption)
- Request quota increases if necessary: Plan for necessary time for quota increase requests to be fulfilled