OPS 2: How do you approach application lifecycle management?

Adopt lifecycle management approaches that improve the flow of changes to production with higher fidelity, fast feedback on quality, and quick bug fixing. These practices help you rapidly identify, remediate, and limit changes that impact customer experience.

Resources

AWS SAM template example using nested stacks
AWS SAM template example using parametrized environment
Quick-start reference for cross-account deployments
Using dynamic references to specify template values in AWS CloudFormation
Amazon Partner Blog: Multi-account deployments

Best Practices:

• Use infrastructure as code and stages isolated in separate environments: Use infrastructure as code and version control to enable tracking of changes and releases. Isolate development and production stages in separate environments. This reduces errors caused by manual processes and helps increase levels of control to gain confidence your workload operates as intended.

• Use CI/CD including automated testing across separate accounts: Automate build, deployment, testing, and rollback of the workload upon KPI and operational alerts. This eases troubleshooting, enables faster remediation and feedback time, and enable automatic and manual rollback/roll-forward should an alert trigger.

• Prototype new features using temporary environments: Use infrastructure as code to create temporary environments for new features you may need to prototype, and tear them down as you complete them. Temporary environments allows for higher fidelity when working with managed services, and increase levels of control to gain confidence your workload integrates and operates as intended.

• Use a rollout deployment mechanism: Use rollout deployment as opposed to all-at-once mechanisms. Rollout deployment can limit application changes to a small set of customers in production gradually while all-at-once is used for non-production systems.

• Use configuration management: Use configuration management systems to make and track configuration changes. These systems reduce errors caused by manual processes, reduce the level of effort to deploy changes, and help isolate configuration to business logic.

• Review the function runtime deprecation policy: AWS provided function runtimes follow official long-term support deprecation policies. Third-party provided runtime deprecation policy may differ from official long-term support. Consider reviewing your runtime deprecation policy and having a mechanism to report on runtimes that if deprecated may affect your workload to operate as intended.

Improvement Plan

Use infrastructure as code and stages isolated in separate environments

Use a Serverless framework to help you execute functions locally, build and package application code, separate packaging from deployment, deploy to isolated stages in separate environments, and support secrets via configuration management systems.

• With AWS Serverless Application Model (AWS SAM), you can use AWS CloudFormation parameters and stacks to parametrize environments.
• With AWS IAM Roles, AWS SAM can assume temporary credentials and deploy your Serverless application to separate AWS accounts.
• AWS System Manager Parameter Store and AWS Secrets Manager have native integration with CloudFormation, and via parameters you can separate dynamic configuration from your infrastructure logic.

For a large number of resources, consider breaking common functionalities such as alarms into separate infrastructure as code templates.

• Use AWS CloudFormation nested stacks to help you deploy them as part of your Serverless application stack.
  Amazon Partner Blog: Amazon CloudFormation Nested Stacks Primer
  Serverless Application example with Nested stacks

Use CI/CD including automated testing across separate accounts

Use a Continuous Integration/Continuous Deployment (CI/CD) pipeline solution that deploys multiple stages in isolated environments/accounts.

Automate testing including but not limited to unit, integration, and end-to-end tests.

Favor rollout deployments over all-at-once deployments for better resilience, and gradually learn what metrics will best determine your workload’s health to appropriately alert on.

Use a deployment system that supports traffic shifting as part of your pipeline, and rollback/roll-forward traffic to previous versions if an alert is triggered.
Traffic shifting using AWS CodeDeploy, AWS SAM and Amazon CloudWatch Alerts

• Examples

  • The order service pipeline automates unit, integration, and end-to-end tests. Deployments are rolled out to customers linearly, and are automatically rolled back should an impact to Order KPIs be observed via alerts.

Prototype new features using temporary environments

Use a Serverless framework to deploy temporary environments named after a feature.

• With AWS SAM or a trusted third-party framework, deploy an AWS CloudFormation stack named after a feature you are working on (for example, feat-new-basket), and tag appropriately to help identify the owner.
  AWS Multiple Account Billing Strategy
  Best practices to secure a newly created AWS Account

Implement a process to identify temporary environments that may have not been deleted over an extended period of time.

• Identify how long a temporary environment generally remains available.
• Include the stack’s owner contact details in AWS CloudFormation stack tags.
• Use Amazon CloudWatch Events scheduled tasks to notify and tag temporary environments for deletion, and provide a mechanism to extend its deletion should that be necessary.

Prototype application code locally and test integrations directly with managed services.

• With AWS SAM or a trusted third-party framework, you can iterate on an AWS Lambda functions’ code locally.
• Use temporary credentials before invoking functions locally so that your application code can interact with a deployed managed service rather than mocks.
• Similarly, run integration and end-to-end tests against a deployed environment via a Continuous Integration pipeline.
  Invoking AWS Lambda functions locally with AWS SAM
  Invoking AWS Lambda functions locally with Serverless framework
  Amazon Partner: Develop locally against cloud services with Stackery

Use a rollout deployment mechanism

For production systems, use a linear deployment strategy to gradually roll out changes to customers.

• Publish a new version for every new AWS Lambda function deployment.
• Create Lambda aliases pointing to versions that reflect features or stable.
• Use aliases for invoking Lambda functions, omitting will invoke the latest application code deployed that may not reflect a stable version or a desired feature.
• Use Lambda alias’ routing configuration to enable traffic shifting by pointing to multiple versions.
  • Examples

    • Lambda alias named stable can be configured to point to application version 1.2 for 90% of the invocation it receives while 10% to be routed to 1.3.

• AWS SAM natively integrates with AWS CodeDeploy to provide rollout deployments for Serverless resources. It supports both linear, canary and all-at-once strategies.
  AWS SAM rollout deployment configuration

For high volume production systems, use a canary deployment strategy when you want to limit changes to a fixed percentage of customers for an extended period of time.
Setting up canary deployments for Amazon API Gateway

Use configuration management

Use environment variables for configuration options that change infrequently such as logging levels, and database connection strings.
Setting up Lambda environment variables
Storing sensitive information in environment variables
Passing API Gateway stage-specific metadata to Lambda functions

Use a configuration management system for dynamic configuration that might change frequently or contain sensitive data such as secrets.
Amazon Partner Blog: AWS Lambda and Secret Management

Review the function runtime deprecation policy

Identify and report on runtimes that might deprecate, and their support policy.

• Review AWS Lambda runtime support policy to understand deprecation schedule for your runtime.
• Review AWS Personal Health Dashboard (AWS PHD) notifications. With AWS PHD, you can also automate notifications and send to custom communication channels other than your AWS Account email.
• Use AWS Config to report on AWS Lambda functions runtime that might be near their deprecation.
• In the event of being unable to migrate to newer runtimes within the deprecation schedule, use AWS Lambda Custom Runtime feature as an interim solution.
  Running compliance and operational checks with AWS Config for AWS Lambda functions
  Automating AWS Personal Health Dashboard with custom notifications
  Bringing your own runtime with AWS Lambda