testing-use-expect-mox

Testing in Elixirland (use expect rather than stub)

Paulo Gonzalez

testing, elixir, mox, expect, stub

I've finally been able to put together a few thoughts I've had into something that may help some folks in the future. At least it will help myself, since I can link to this post when this comes up again.

After working in a few large Elixir codebases, I've noticed a recurring pattern: lingering stub/3 calls that can safely be removed from tests without changing behavior. When removed, nothing fails. More importantly, there's a missed opportunity to use expect/3, which in most cases would have been the right tool.

The stub is a mute button. A test that used to tell you that a contract was to be respected (an implementation would be called at some point) no longer tells you anything. In fact, it tricks you into thinking that it does, but it doesn't.

Even by using setup :verify_on_exit!, nothing happens. We still get the misdirection and dead code in the codebase because stub/3 allows things to be called zero times. The best thing about Mox is verifying that the contract is exercised. The moment it is no longer exercised, I'd want tests to warn me about it.

The false negative here is what bothers me the most: I don't know that something STOPPED being called. These are annoying and usually cause surprises.

From what I've seen, folks reach for stub/3 because it's an overloaded term in testing, much like "mock" (as shown in this excellent Dashbit article). Folks are well-meaning, but the end result seems to always be the same:

The reasons for this happening are mostly:

- The term "stub" feels familiar from other libraries and languages. Many users don't immediately notice or understand expect/3 as the verification-based alternative and how powerful it is.

- People often come from other ecosystems where the word "stub" has a different meaning than in ours. Once I show how a certain stub call can be removed and mention expect/3, it tends to click for them.

The best thing about expect/3 is that it verifies the contract is exercised. When a contract stops being exercised, your tests will immediately fail and warn you. This is exactly what you want: early feedback when something changes.

With stub/3, you lose this verification. The test might pass, but you've lost valuable information about whether your code is actually using the dependency you thought it was.

I added some docs to Mox to try to communicate this. Here is the Mox Github issue discussing these symptoms and here is the PR that addresses it.

Reach for expect/3 by default. It verifies your contracts are actually being exercised and will warn you immediately when they stop being used. Save stub/3 for the rare cases where you truly need zero-or-more-times semantics.

Your tests should be a safety net that catches when things change, not a mute button that hides what's happening.