For a development team it is critical to have a prioritized list, with tight feedback cycles, while mitigating the right risks. This article is going to focus on the second part of that.
We want short feedback loops to enable development teams to make changes at a healthy and sustainable pace. It is important not to go slow (boring! stagnant!) and to avoid boon doggles, but at the same time we don't want to be the overconfident downhill skier that runs into a tree or the irresponsible driver that slides off the road into a ditch because they ignored the slick road conditions on a dark rainy night. We want to make small course corrections early and quickly: measure, adjust, repeat.
To do that, we need to create an environment which focuses on achieving outcomes while enabling lots of small experiments and autonomy. That puts a team in the best possible position to react to planned (release/change based) and unplanned (external events) feedback.
To enable lots of small experiments at a sustainable pace, it is critical to minimize surprises and maximize predictability in the areas that are tangential to an intended change/experiment. However unknown unknowns will always exist, so it is important to counter-balance treading too lightly with tools and practices that allow inspecting and adapting, backing out and fixing forward for the things we will inevitably fail to predict. This allows you to lean in with some justifiable confidence, but also not let perfection be the enemy of progress.
A number of base practices enable this at the individual and team level across the areas of application, infrastructure, and releases/testing:
- Business Rules are codified as source code. Source code is checked in to version control for collaboration and to enable rollback to prior state. Good commit and PR comments facilitate learning as a team while the system evolves. Version Control.
- Source code has a reasonably balanced cognitive load requirement for the domain, enabling modifications by team members with varying familiarity and experience levels. Prudent investments in automated tests around the most impactful application logic enable its [relatively] safe modification. Software Architecture, Clean Code, and automated tests.
- Application builds are reproducible. The scripted processes which build an application run on any machine, consistently. That build process is checked in to version control. Reproducible builds, Version Control.
- Application code builds continuously. Local feedback cycles are quick, small change-sets are pushed to version control and everything that is committed to version control is built and tested/checked with automation. That automtaion protects shared environments and sources of truth such as master/release branches. Continuous Integration
- Application configuration for each environment is in version control or an audit-able configuration management tool. Secrets are secured appropriately. When it comes to question of how something is configured different in one environment vs another, nothing is left of imagination. The whole team has a mechanism to inspect. Team ownership of deployment, DevOps.
- Deployment to the highest promotion environment possible is done automatically. That's possible as a result of following the rest of these practices. Continuous Delivery
- Application changes are rolled out based on rules (i.e. - Canary / AB test) which enable safety around unknown unknowns. Useful in lower environments, but especially for production because it is unique no matter how much insurance you attempt to put in place. Progressive Delivery and test in production
- There are executable scripted processes to build the infrastructure an application requires. Again, nothing is left to the imagination. Everything about how to make the whole thing work is codified and checked in to version control. Particularly critical here, having this knowledge in code enables building ephemeral 'clone' environments to vet changes before they are promoted. Infrastructure as Code
- Manual modifications to Infrastructure are actively prevented or discarded. To modify Infrastructure in a manner that will stick requires a code change in source control. In conjunction with the other practices listed, this forces knowledge that might otherwise be hidden, to be committed to source control. Immutable Infrastructure: faster to scale, more secure, predictable, and understood.
- Upgrades and changes to infrastructure and server configuration can be vetted and released without significant downtime or maintenance windows. Scheduling a maintenance window or having to schedule around low-traffic periods hinders a feedback loop. High Availability is enabled with load balancer rules capable of shifting traffic to new/modified servers, either rolling through them or spinning up dark clusters (i.e. - red/black), vetting them and cutting over as they prove they are behaving as expected.
- Source control is the source of truth for every change from Infrastructure to app code/config and is automatically applied from master or per environment branches. These branches are protected from bad changes by PR checks which prevent merges, often vetting things in ephemeral environments. GitOps
- Ability to gather detailed information around the current and recent state of production via self service tools for: logging, metrics, traces. Observability, DevOps.