OPS 6: How do you mitigate deployment risks?

Adopt approaches that provide fast feedback on quality and enable rapid recovery from changes that do not have desired outcomes. Using these practices mitigates the impact of issues introduced through the deployment of changes.

Resources

Deep Dive on Advanced Continuous Delivery Techniques Using AWS
AWS Developer Tools
AWS CodeDeploy User Guide
Try a Sample Blue/Green Deployment in AWS CodeDeploy

Best Practices:

• Plan for unsuccessful changes: Plan to revert to a known good state, or remediate in the production environment if a change does not have the desired outcome. This preparation reduces recovery time through faster responses.

• Test and validate changes: Test changes and validate the results at all lifecycle stages to confirm new features and minimize the risk and impact of failed deployments.

• Use deployment management systems: Use deployment management systems to track and implement change. This reduces errors cause by manual processes and reduces the effort to deploy changes.

• Test using limited deployments: Test with limited deployments alongside existing systems to confirm desired outcomes prior to full scale deployment. For example, use deployment canary testing or one-box deployments.

• Deploy using parallel environments: Implement changes onto parallel environments, and then transition over to the new environment. Maintain the prior environment until there is confirmation of successful deployment. Doing so minimizes recovery time by enabling rollback to the previous environment.

• Deploy frequent, small, reversible changes: Use frequent, small, and reversible changes to reduce the scope of a change. This results in easier troubleshooting and faster remediation with the option to roll back a change.

• Fully automate integration and deployment: Automate build, deployment, and testing of the workload. This reduces errors cause by manual processes and reduces the effort to deploy changes.

• Automate testing and rollback: Automate testing of deployed environments to confirm desired outcomes. Automate rollback to previous known good state when outcomes are not achieved to minimize recovery time and reduce errors caused by manual processes.

Improvement Plan

Plan for unsuccessful changes

Plan for unsuccessful changes: Plan to revert to a known good state (that is, roll back the change), or remediate in the production environment (that is, roll forward the change) if a change does not have the desired outcome. When you identify changes that you cannot roll back if unsuccessful, apply due diligence prior to committing the change.

Test and validate changes

Test and validate changes: Test changes and validate the results at all lifecycle stages (for example, development, test, and production), to confirm new features and minimize the risk and impact of failed deployments.
AWS Cloud9
What is AWS Cloud9?
How to test and debug AWS CodeDeploy locally before you ship your code

Use deployment management systems

Use deployment management systems: Use deployment management systems to track and implement change. This will reduce errors caused by manual processes, and reduce the level of effort to deploy changes. Automate the integration and deployment pipeline from code check-in through testing, deployment, and validation. This reduces lead time, enables increased frequency of change, and further reduces the level of effort.
Introduction to AWS CodeDeploy - automated software deployment with Amazon Web Services
What is AWS CodeDeploy?
What is AWS Elastic Beanstalk?
What is Amazon API Gateway?

Test using limited deployments

Test using limited deployments: Test with limited deployments alongside existing systems to confirm desired outcomes prior to full scale deployment. For example, use deployment canary testing or one-box deployments.
Working with deployment configurations in AWS CodeDeploy
Blue/Green deployments with AWS Elastic Beanstalk
Set up an API Gateway canary release deployment

Deploy using parallel environments

Deploy using parallel environments: Implement changes onto parallel environments, and transition or cut over to the new environment. Maintain the prior environment until there is confirmation of successful deployment. This minimizes recovery time by enabling rollback to the previous environment. For example, use immutable infrastructures with blue/green deployments.
Working with deployment configurations in AWS CodeDeploy
Blue/Green deployments with AWS Elastic Beanstalk
Set up an API Gateway canary release deployment

Deploy frequent, small, reversible changes

Deploy frequent, small, reversible changes: Use frequent, small, and reversible changes to reduce the scope of a change. This results in easier troubleshooting and faster remediation with the option to roll back a change.

Fully automate integration and deployment

Use build and deployment management systems: Use build and deployment management systems to track and implement change, to reduce errors caused by manual processes, and reduce the level of effort. Fully automate the integration and deployment pipeline from code check-in through build, testing, deployment, and validation. This reduces lead time, enables increased frequency of change, and reduces the level of effort.
What is AWS CodeBuild?
Continuous integration best practices for software development
Slalom: CI/CD for serverless applications on AWS
Introduction to AWS CodeDeploy - automated software deployment with Amazon Web Services
What is AWS CodeDeploy?

Automate testing and rollback

Automate testing and rollback: Automate testing of deployed environments to confirm desired outcomes. Automate rollback to previous known good state when outcomes are not achieved to minimize recovery time and reduce errors caused by manual processes. For example, perform detailed synthetic user transactions following deployment, verify the results, and roll back on failure.
Redeploy and roll back a deployment with AWS CodeDeploy