I miss make deploy
I spent a few years working at a company. Their deployment process was simple.
make deploy
That’s it. No Jenkins, no Github Actions, no IaaC, nothing. The code was built, tested, containerized and deployed to our cloud hosting provider, usually within seconds. Although, that’s a little disingenuous. The above command would just deploy to our sandbox environment. If you wanted to deploy to prod you had to be a little more serious.
make deploy PROJ=prod
Developers could deploy whenever they wanted.
I remember joining and thinking that that was quite silly. The place I came from wasn’t too crazy but they at least had gitops and a CI/CD pipeline. Then I got used to it. I got into the habit of deploying to prod several times a day. Then I left and worked at a few places since and man do I miss make deploy.
I miss it every damn day. It was fast, simple, and effective. It worked every time. The only thing between me and prod was an active internet connection and my imagination.
Things that made make deploy a success:
every backend service could be independently deployed
every service was deployed via make deploy and make deploy only (with few exceptions)
rolling back was a very quick and easy task
frontend deploys had their own release schedule and process that was independent of backend deploys unless coordination was explicitly required, and that was handled case-by-case.
the devs that worked there had a sense of ownership over their code, services and deployments. The buck stopped with them. No other person or process to blame for their production issue.
I know what you’re thinking. Amateur hour. Of course that worked for a tiny company but we out here in Serious Software Engineering™️ land run these complex deployment pipelines for a reason! We can’t just have developers pushing out code when they think it’s ready. We have reviews! Deployment managers! Maintenance windows! 500 page rollback plans in a google doc somewhere!
You could call it a startup the day I joined but the company was already handling millions of dollars of customers’ money at the time. Said company is now worth billions and has even acquired other companies. But chatting with my old boss there I learned that make deploy endures. It warms my heart.
There were some half written rules, of course. Whatever is running in prod should be on the main branch and go through code review. Features should be tested out in our sandbox environments first. This was all easily auditable. make deploy knew how to label containers and code versions so anyone would know what’s running out there. Sure, someone could write some code and deploy it without anyone’s consent. It happened once in a while. Sometimes accidentally. Sometimes in a roll-forward-and-worry-about-committing-the-changes-later scenario. Never in a I’m-trying-to-be-sneaky way as far as I know. It would be easy to tell.
I’m wracking my brain and I couldn’t remember any issues that came up because the deployment pipeline was too simple when I worked there. I think I once accidentally deployed something to prod from a feature branch, but the feature was new, the tests passed, and the deployment succeed and no alerts fired so no actual harm was done. And even if there was a problem I knew rollbacks could happen in seconds. Yolo.
And yet I can immediately think of tons of problems and outages caused by overly complex deployments. Release managers mixing up commit hashes. Problem in service A causes release process to be halted halfway through, leaving prod in a semi-deployed state and causing an outage. Trying to roll back the entire thing causes yet more issues because a dev merged a backwards incompatible database migration 3 weeks ago and is now hiking Patagonia without a cellphone. And now Terraform Cloud is down. Let’s just try again tomorrow. Uh oh. Now it’s Github’s day off.
Not to mention just the general snails pace of development in places like that. Code I write today may not be deployed for weeks. Who knows how it’ll interact with the state of things by then. And it’s probably not being released by me, or maybe it is, but I’m also releasing 20 other peoples’ code at the same time. A lot of that code was written today because they know the next release window isn’t for another 2 weeks. Pray for me. Also it’s 8pm on a Thursday because an outage we had a few years ago requires all releases be done outside of business hours. If this doesn’t go well, no sweat I guess, we have a backup window on Saturday. Thank god.
I’ve learned two things in my career:
bad code makes it into prod no matter what
You can put all the bureaucracy in the world between devs and your production environment but issues are always, say it with me, always going to slip through. But since you already have tons of processes and runbooks and managers and a “blameless” culture you get to spend the next week talking through how those things failed and never admit the real issues. Maybe a 3rd approver on each PR is the move.
teams that deploy more often write better code, better features and fuck up way less often
Pull requests become smaller and easier to review, which makes deployments much smaller, which makes prod issues much easier to trace, which means bug fixes go out faster, which hardens the entire platform, which allows devs to gain confidence when deploying, which speeds up development, which means more feature releases. And on and on.
Don’t release. make deploy.