OPS 6: How do you mitigate deployment risks?

Adopt approaches that provide fast feedback on quality and enable rapid recovery from changes that do not have desired outcomes. Using these practices mitigates the impact of issues introduced through the deployment of changes.

Resources

Deep Dive on Advanced Continuous Delivery Techniques Using AWS
AWS Developer Tools
AWS CodeDeploy User Guide
Try a Sample Blue/Green Deployment in AWS CodeDeploy

Best Practices:

Improvement Plan

Plan for unsuccessful changes

  • Plan for unsuccessful changes: Plan to revert to a known good state (that is, roll back the change), or remediate in the production environment (that is, roll forward the change) if a change does not have the desired outcome. When you identify changes that you cannot roll back if unsuccessful, apply due diligence prior to committing the change.
  • Test and validate changes

  • Test and validate changes: Test changes and validate the results at all lifecycle stages (for example, development, test, and production), to confirm new features and minimize the risk and impact of failed deployments.
    AWS Cloud9
    What is AWS Cloud9?
    How to test and debug AWS CodeDeploy locally before you ship your code
  • Use deployment management systems

  • Use deployment management systems: Use deployment management systems to track and implement change. This will reduce errors caused by manual processes, and reduce the level of effort to deploy changes. Automate the integration and deployment pipeline from code check-in through testing, deployment, and validation. This reduces lead time, enables increased frequency of change, and further reduces the level of effort.
    Introduction to AWS CodeDeploy - automated software deployment with Amazon Web Services
    What is AWS CodeDeploy?
    What is AWS Elastic Beanstalk?
    What is Amazon API Gateway?
  • Test using limited deployments

  • Test using limited deployments: Test with limited deployments alongside existing systems to confirm desired outcomes prior to full scale deployment. For example, use deployment canary testing or one-box deployments.
    Working with deployment configurations in AWS CodeDeploy
    Blue/Green deployments with AWS Elastic Beanstalk
    Set up an API Gateway canary release deployment
  • Deploy using parallel environments

  • Deploy using parallel environments: Implement changes onto parallel environments, and transition or cut over to the new environment. Maintain the prior environment until there is confirmation of successful deployment. This minimizes recovery time by enabling rollback to the previous environment. For example, use immutable infrastructures with blue/green deployments.
    Working with deployment configurations in AWS CodeDeploy
    Blue/Green deployments with AWS Elastic Beanstalk
    Set up an API Gateway canary release deployment
  • Deploy frequent, small, reversible changes

  • Deploy frequent, small, reversible changes: Use frequent, small, and reversible changes to reduce the scope of a change. This results in easier troubleshooting and faster remediation with the option to roll back a change.
  • Fully automate integration and deployment

  • Use build and deployment management systems: Use build and deployment management systems to track and implement change, to reduce errors caused by manual processes, and reduce the level of effort. Fully automate the integration and deployment pipeline from code check-in through build, testing, deployment, and validation. This reduces lead time, enables increased frequency of change, and reduces the level of effort.
    What is AWS CodeBuild?
    Continuous integration best practices for software development
    Slalom: CI/CD for serverless applications on AWS
    Introduction to AWS CodeDeploy - automated software deployment with Amazon Web Services
    What is AWS CodeDeploy?
  • Automate testing and rollback

  • Automate testing and rollback: Automate testing of deployed environments to confirm desired outcomes. Automate rollback to previous known good state when outcomes are not achieved to minimize recovery time and reduce errors caused by manual processes. For example, perform detailed synthetic user transactions following deployment, verify the results, and roll back on failure.
    Redeploy and roll back a deployment with AWS CodeDeploy