The Costly Mistakes Fintechs Make When Trying To Improve Security

Time to call a DevOps engineer to get permission or have them run the patch.

The worst thing that can happen to an engineer is to get paged out-of-hours only to realize they can't fix the problem on their own. They did all the hard work of waking up, debugging the problem, and finding a solution. But when it's time to apply the fix, they don't have access. Time to call a DevOps engineer to get permission or have them run the patch.

DevOps means developers run their own code, but how can a developer operate a piece of software if she can't access the database, the cloud provider, or the Kubernetes cluster? Only a handful of people has access to these resources at most companies today.

This problem is not just bad during out-of-hour pages. DevOps teams centralizing raw access to production are bottlenecks to the whole engineering team. A simple query in the database to troubleshoot a problem can take hours for the busy and sad DevOps team to process the request in their queue.

It's not ok to keep making direct updates to the database or change things in the AWS console all the time. CI/CD and infrastructure as code are great tools. But direct access will happen no matter how much automation a company has. Restricting raw access to a few engineers results in bad culture incentives and an environment with low trust and autonomy.