Back in the days, when Microsoft built Windows Vista, they relied on hundred of thousands of GUI based end-to-test tests that ran for days for each build. Fast forward 15 years, end-to-end tests are despised for being slow and flaky. At Facebook, there is a huge push back from the engineers to write end-to-end tests as they prefer unit tests and integration tests for being fast and easy to write. While we all know of the testing pyramid which beautifully shows that while a lot of unit tests is what we need the most, a good number of integration tests and few end-to-end tests are also needed to strike the right balance.
If we try to find the intrinsic motivation of engineers shying away from end-to-end and to some extent, even integration tests is not that they don’t see the value in it but they are just hard and cumbersome. That’s not a good enough excuse to not write these tests. Common sense tells us that we will pay an even bigger price for not having the right tests in our arsenal in right proportion. Mark famously said “Move fast and break things” and has now evolved that mantra to “Move fast with Stable Infrastructure” to make the point.
Here comes Test Infrastructure, the elephant in the room. You may even ask, what the heck is test infrastructure? I like to define test infrastructure is providing tools, frameworks and support needed to engineers so they can do what they do best i.e. write code and write tests to test that code without having to deal with hard problems like test setup, test users, test environments, test data, test frameworks, scheduling tests at different development stages, test selection, test monitoring and retries, bisect (identifying blamed code when a test breaks), identifying flaky tests and the list goes on.
The talk is going to be about how an organization can build necessary test infrastructure to support engineers to provide high quality signal and reduce the noise that’s associated with writing and running hundred and thousands of tests every 20 mins. While there are a lot of DevOps tools that claim to solve this problem but we still need test infra specialists in each of these organizations to build what’s missing and make it easy for engineers to ship high quality code without having to deal with test infrastructure related issues.