Introduction

A few years ago, we had a very “traditional” development model here at ING Bank Spain. Given the relatively small number of people working at any given project and the stale state of technology in the banking business this wasn’t a big deal. The development lifecycle was composed of three major environments (aptly named Development, Preproduction and Production). With the growth of our business, the introduction of new challengers in the banking industry and, above all, the need to keep up with our customers’ expectations, there was simply no way to deliver more and better software without increasing the headcount.

Textbook waterfall

Textbook waterfall

You can’t just hire every developer on the street and expect everything to stay the same. The existing way of working and old habits quickly start crumbling under the weight of new people. What should we do? Hire more managers? More synchronization meetings? We opted for Agile (more specifically Scrum), moved people around a bit and flattened the hierarchy in an effort to maintain (if not increase) the efficiency.

That also meant that 3 environments were no longer enough. There’s a very big issue with having this kind of “centralized” environments while also trying to do Continuous Integration (Yegor <explains this http://www.yegor256.com/2014/07/21/read-only-master-branch.html> beautifully), and it comes down to having everyone just waiting around whenever a bug causes a crash in the centralized testing  environments.

So we heard our colleagues in Australia had faced and solved a similar issue and we went there to see exactly how. Enter “Bank in a Box”…

The Concept

The idea behind all of that was to take the whole bank that we have in production, downsize it to manageable levels and virtualize everything. I should mention that production data (yes, all the data is also inside BIAB) was also asymmetrically masked (meaning we can’t “unmask” it) with parts of it being ciphered – this ensures regulatory compliance and protects our customers’ privacy.

So we took the whole technology stack, brought it to Spain, and started integrating the platform with our existing tools. Now testing your code was only a matter of committing your changes to the version control and pushing one button to deploy those changes into your own environment – even if it breaks, only a couple of people are now affected so there’s lots of freedom to experiment.

bankinabox2

Lessons learned

Building your own private cloud is hard. There’s thousands of blog posts about it, so no need to go into detail. While it’s usually a mistake to build your own private cloud in this day and age, the banking industry is heavily regulated and has very strict requirements. Being engineers, we chose to spend our time building our own instead of fighting the regulators.

The second challenge is the operational side of it – where we once had a few dozen machines to manage, we now have thousands. Point and click only goes so far and tons of automation was required to ensure everything is working smoothly.

The third, and probably the biggest one, is that giving a copy of the bank to our development teams and thinking everything would run smoothly was too optimistic. Soon, the dreaded cry of “my apache stopped working, how can it be fixed?” (hint: it’s probably not the apache) echoed through our halls.

We came from a culture where a developer only develops and it’s the system administrator’s job to ensure the service stays up and running, now we were forcing ourselves to transition into something similar to DevOps – not true DevOps since, in production, root access is restricted.

Conclusions

No single point of failure helped us reduce the bottlenecks in the development process. There are dayto-day challenges related to maintaining such a big infrastructure, but overall the quality of life improved immensely for our teams. Although it’s not cheap, it helped us actually improve the efficiency of our teams in delivering value to our customers, so there’s a pretty significant net gain.

Greater freedom (everyone has root in the BIABs) is helping our development teams grow a culture which is much more transparent from the technical side. While there’s still bits of magic here and there, everyone is now free to open an SSH connection, explore, read the logs and tinker with a system which is very close to the actual production environment.

Not affecting other teams means that, as a developer, you’re now much less scared into pushing big changes – although not a perfect clone, the similarity to production itself helps us gauge the side effects much more accurately before going through with them. This not only boosts development speed, it also helps everyone sleep a little better at night.