In “Growing Object Oriented Software Guided By Tests”, Steve Freeman and Nat Pryce talk about the dangers of tests that occasionally fail, otherwise known as flickering tests. These failures can cause teams to start seeing these failures as false positives, and distrust their build results. I know – it’s happened to me, especially with end-to-end Selenium tests.

We experienced some test flickering in our Selenium builds, and guess what: they were real bugs. Obscure, tricky to find bugs that happened only occasionally. But the Selenium tests ran dozens of tests after every commit, which meant the failure turned up frequently in the build. The benefit was in frequency of using the system.

For me, most of these flickering failures have happened in end-to-end tests. These can be the trickiest to get right, as they have to coordinate interaction between many components of a system – often in an asynchronous manner.

There has been a lot of discussion about this topic recently.

• Freeman and Pryce address this in their book, with some strategies and tools for dealing with asynchronous interaction. If you haven’t read “Growing Object-Oriented Software Guided By Tests” (you’ll see this book referenced on Twitter via the acronym homophone #goos), I highly recommend it. The authors know a lot about good design, and you’ll experience it first-hand as they build a working example from start to finish.
• Brian Marick has started reading through #goos and is translating the examples into Ruby. He’s been blogging about his experiences as well, including some of his thoughts about end-to-end tests.
• Steve has responded to Brian, and ends with a thought provoking question “What would it take to make automated end-to-end tests less scary?”

How have you dealt with flickering test failures? Were they end-to-end tests?