Unlocking speed and autonomy with refactoring

In this article, we share lessons learned from refactoring large swaths of our code so that you can avoid some of the pitfalls involved, and of course reap the benefits of having code that works well and is easy to change.

One of the eight principles ofthe Adyen Formulais “we launch fast and iterate”. Speed is an integral part of how we work. We want to always keep pace with technology and our customers’ ambitions. Owing to the rapid expansion of our engineering organization, and increasing complexity of our software stack, we have had quite the work cut out for us taming said complexity.

In response to these challenges, this year, we decided to prioritize empowering development teams at Adyen to deploy their code to production with speed and autonomy, while maintaining the level of security and reliability that our customers expect from us.

We identified a number of high-level objectives that would enable us to achieve this goal, such as decentralizing our deployment process and significantly increasing the quality of our code.

This company-wide initiative couldn’t have come at a better time for my team, the Recurring and Tokenization team, which owns several heavily used products on the Adyen platform includingAdyen Tokenization,Network Tokenization,Account UpdaterandAuto Rescue.

We were already experiencing the growing pains of dealing with a complex codebase worked on by hundreds of engineers: as evidenced by how challenging it was for us to make certain changes to the code, but also in terms of a few incidents we experienced.

We decided to refactor several parts of our code with focus on specific areas that, when improved upon, would get us where we wanted to be in terms of development velocity and product reliability:

We wanted to simplify some of the more complex business logic in our code so they are easier to understand but also more testable. In addition to this, we wanted to clean up our dependency tree in order to speed up our builds which would result in a better development experience and save CI/CD pipeline resources.

Our main goal with containerization was to decouple our application from the infrastructure it runs on.

This has some nice benefits such as facilitating fully automated deployments, reproducibility (the same software artifact is promoted from staging to production), and effective resource allocation/utilization (smaller apps are allocated fewer resources and larger apps don’t starve the smaller ones).

It’s one thing to have a single button you can just click to ship your code to production. It’s another to be confident that your system won’t go up in flames immediately afterwards.
You need comprehensive monitoring (we’ve spent a couple of months expanding and fine-tuning our monitoring systems). You also need tests… lots of it! We refactored our code to make it more testable and of course added more tests.

While we’re certainly not there yet, we’ve made a lot of progress in the last quarter and are excited for what lies ahead; the productivity we’ll be unlocking for ourselves, and the features we’d be able to deliver reliably to our customers.

Should you also be refactoring your code? The answer is yes! Refactoring is inevitable and a continuous process.

At some point, you'd need to:

Every now and then, there is a trade-off made when shipping a feature quickly at the expense of maintainable or scalable code. These debts have to be paid sooner or later if you want to maintain the quality of your product and/or make it quick and easy to iterate upon.

The complexity in code is a result of the complex world we live in. Business requirements are constantly changing – you need to keep up with the competition and innovate by shipping new products and features, grow your revenue by expanding to new industries and markets, comply with new regulations etc.

Eventually, the assumptions made in the code about how things work would become false and you’d need to refactor it to conform to the new reality.

Software, even feature-complete ones, gather cruft over time. You might need to refactor your code in order to fix security vulnerabilities, handle the scale of current or future workloads, unlock better performance from updated/new dependencies, or migrate from a software or hardware platform that is no longer supported.

Let’s dive into the lessons we learned on our refactoring journey.

It’s very important to dedicate time to refactoring. Itcan’tbe an afterthought, otherwise more seemingly pressing issues will get in the way.

It’s also a continuous process. There’s no start or end date. It’s like vacuuming your house; you have to do it regularly otherwise your house would eventually become a dumpster!

On my team, we started with “fix it” days (nicknamed “coding dojo”). Fix it days are one day every month when we organize in groups of two (2) or more engineers to work on prioritized technical debt tickets.

This evolved into “10% time”; one day every two weeks when engineers can work solo or collaborate on a problem of their choosing that reduces technical debt in our code, or improves the quality of our products. In addition to this, we try to allocate time in each sprint for a few small refactoring tasks.

You can’t fix everything at once, so you should tackle the most important problems first. What are some recurring complaints you hear developers make about working with your codebase? What source files are most frequently changed or were involved in the incidents that occurred in the recent past? Perhaps you could start with the issues you identify by thinking through these questions.

We’ve found incident retrospectives a good avenue to seek areas in our system that need improvement. We have also utilized pre-mortems, which enable us to be proactive rather than reactive.

There’s always risks involved in making changes to your code, especially significant ones in business-critical flows. It’s important to understand what you’re changing and what could happen if something were to go wrong. Then come up with strategies to mitigate potential bad outcomes.

Here are a few things that helped us:

Small changes means faster and more effective reviews. Your reviewer would be better able to catch potential issues in your code. Also, should your changes end up breaking something, it’d be easier to revert. And you only lose a small part of your work, not the whole refactor.

Write tests for the code first if it has little or no tests before refactoring it. You’re less likely to break things this way.

Wrap your new code in a feature toggle. If your code breaks something, the toggle makes it quick and easy to go back to the old version while you fix the issue at your convenience.

Run both old and new versions of the code for some time but only use the result of the old one. Compare the two versions to find discrepancies. If the old code disagrees with the new one, you probably have a bug somewhere which you have discovered without breaking anything and can investigate/fix later. When everything looks good, you can remove the old code.

You should factor in the performance implication of running the computation you’re refactoring twice. If the impact is significant, you probably want to avoid this strategy or use it for just a small subset of your workload.

An easy way to rack up technical debt is by ignoring anti-patterns in your codebase. All it takes is a few instances of these code smells, and suddenly they begin to proliferate uncontrollably because developers want to save some little time or might not even be aware that they are problematic.

Taking a systematic approach by identifying and preventing anti-patterns means you can enjoy faster development, deploy fewer bugs and have less refactoring to do in future.

We’ve discussed common anti-patterns as a team, found examples in our codebase, and agreed to stop using them. We also came up with plans to remove existing usages, and look out for them in code reviews. Furthermore, we employ static code analysis tools and quality gates in our CI pipelines to help us identify some of them and prevent regressions.

In this article, we stressed the importance of refactoring and delved into how approaching it systematically is helping us deploy code faster and more reliably. We also gave you some insights on why you should and how you can approach refactoring. Hopefully, you have more clarity and can apply our learnings to your codebase.