Uncovering Undocumented Behavior: The Power of Characterization Tests

Here's a common, often uncomfortable truth about software that's been in production for a few years: your existing regression tests likely describe what the system should do, based on someone's initial intentions. They rarely capture the full spectrum of what the system actually does. This distinction is critical because, over time, production systems accumulate a layer of unplanned, undocumented behaviors—quirks, edge cases, and defaults—that users or other services quietly come to depend on.

It's in this gap between intended and actual behavior that many of the most frustrating regressions lurk. Traditional, intent-based tests are blind to these implicit dependencies. Thankfully, there's a specialized testing approach designed precisely for this scenario, though it remains surprisingly underutilized.

The Unwritten Contract: Hyrum's Law in Action

Most developers are familiar with Hyrum's Law, coined by Google's Hyrum Wright: "With a sufficient number of users of an API, it does not matter what you promise in the contract: all observable behaviors of your system will be depended on by somebody." This isn't just about documented API contracts; it encompasses every observable aspect. Think about the order of results, the exact phrasing of an error message, the response time, or whether an empty field returns null or is omitted entirely.

If you've ever maintained a widely used service, you've likely experienced this firsthand. You fix what seems like an obvious bug, only to receive a report an hour later that you've broken someone's integration built on that very "wrong" behavior. From their perspective, you introduced a regression. Standard regression tests, focused on explicit requirements, simply can't safeguard against these implicit dependencies.

Characterization Tests: Capturing Reality

Michael Feathers, in Working Effectively with Legacy Code, gave a name to the solution: characterization tests. The core idea is brilliantly simple: before you modify unfamiliar code, you write tests that capture its current behavior. You're not asserting what it should do, but what it does right now.

The process involves observing the code's output for various inputs and then pinning that observed behavior in a test. If you uncover what looks like a bug, you pin it anyway, perhaps with a note. The objective isn't to validate correctness; it's to create a safety net that immediately flags any behavioral shift. This requires a mental flip: instead of dictating desired outcomes, you're establishing a baseline of reality, ensuring any future change is a deliberate decision, not an accidental byproduct.

Overcoming Resistance: Why They're Not Common

Despite their utility, characterization tests aren't a common practice, largely due to human factors:

• The Slog: Manually characterizing a complex function means meticulously noting outputs for a range of inputs. For an entire service, spanning multiple endpoints, edge cases, and downstream interactions, this becomes a monumental, time-consuming effort.
• The "Feels Wrong" Factor: Writing a test that asserts behavior you suspect is buggy goes against a developer's natural instincts. It demands a conscious effort to understand that pinning the bug is the very point: you're ensuring that if that "bug" ever changes, it's a consequence of an intentional decision.

Tools for the Job: Golden Masters and Snapshot Tests

To mitigate the "slog," techniques like golden master testing (also known as approval testing) and frontend snapshot tests have emerged. These approaches capture a large block of output—be it a rendered UI component, a report, or a function's return value—and save it as an approved baseline. Subsequent test runs compare new output against this baseline, flagging any discrepancies. This significantly reduces the overhead of individual assertions, making characterization more efficient for components that can be called directly.

However, these methods become less straightforward when characterizing an entire distributed service. A service's behavior isn't just its direct return value; it includes its reactions to real requests, the database queries it executes, the messages it sends to queues, and the calls it makes to other services.

Characterizing Services with Real Traffic

This is where the true power of characterization tests for services lies: using real production traffic as the ultimate behavioral specification. Every live request and response pair is, in essence, an unwritten characterization test. By capturing inbound requests and outbound responses, you document the service's behavior for inputs it actually receives, including those odd edge cases nobody would manually invent.

Taking this further, if you can also capture the service's interactions with its dependencies—database queries, messages to message brokers, HTTP calls to other microservices—you've pinned its crucial side effects. Recording this at the network level allows for comprehensive capture without modifying application code. When these recordings are replayed, with dependencies mocked from the same capture, you create a hermetic, repeatable characterization suite. Any deviation—a different response, an unexpected database call, or a changed message—is highlighted in a diff. This is particularly valuable for detecting subtle, expensive regressions where the external response remains unchanged, but internal operations, like a new, inefficient query, have drastically altered.

Navigating Characterization Test Failures

Characterization tests behave differently than traditional unit or integration tests, which can take some getting used to. A failing characterization test doesn't automatically mean "bug"; it simply means "behavior changed." The immediate follow-up question is always: "Was this change intentional?" Sometimes, the answer is yes—you deliberately fixed an old quirk, perhaps after coordinating with affected consumers, and the new behavior becomes the new baseline. Other times, the answer is no, indicating you've just caught a regression that no intent-based test would have identified.

This process fosters a healthy team habit: behavioral changes evolve from accidental occurrences to deliberate decisions, often sparking valuable discussions during pull request reviews. To keep these tests useful, managing noise is essential. Dynamic elements like timestamps and generated IDs will cause constant diffs; these fields should be specifically ignored, but care must be taken not to over-ignore, preserving the integrity of broader assertions.

When Characterization Tests Truly Shine

Several scenarios particularly benefit from this testing approach:

• Refactoring Legacy Code: This is the classic use case Michael Feathers described. By pinning existing behavior, developers can confidently restructure and improve code without fear of introducing unknown regressions.
• Framework and Library Upgrades: Major version bumps in frameworks or ORMs can subtly alter behaviors (e.g., serialization formats, result ordering) that intent-based tests often overlook. Characterization tests catch these nuances.
• System Rewrites: When porting a service to a new language or architecture, the recorded production traffic of the old system becomes the definitive specification for the new one. Differences highlight either deliberate improvements or regressions, eliminating arguments over "intended" behavior.
• AI-Assisted Changes: With AI increasingly drafting large refactors and accompanying tests, an independently captured baseline provides a critical, unbiased verification layer, ensuring the AI's changes don't introduce unseen issues.

Conclusion

Every mature codebase effectively operates with two specifications: one encoded in its traditional tests, reflecting past intentions, and another living in its observable behavior, shaped by years of production usage. Most regression testing solely protects the former, leaving systems vulnerable to issues arising from the latter. Characterization tests bridge this gap.

Ready to experience this firsthand? Pick a service you're hesitant to modify. Before your next change, capture its actual behavior, run your modifications against that baseline, and meticulously review every detected difference. It's highly probable you'll uncover a behavior you never knew existed—and it's far better to discover it in a diff than through an urgent bug report from an unhappy user.

FAQ

Q: How do characterization tests differ from traditional unit or integration tests? A: Traditional tests assert what code should do, based on requirements or design. Characterization tests, however, assert what the code does right now, regardless of whether that behavior is ideal or intended. Their primary goal is to act as a safety net against unintended behavioral changes, not to validate correctness against a specification.

Q: What are the main challenges in implementing characterization tests for a large service? A: The primary challenges are the effort required to initially capture and maintain the behavioral baseline (the "slog") and the need to filter out noise from dynamic data (like timestamps or generated IDs) to keep the diffs meaningful. Additionally, teams must adapt their mindset to treat a failing characterization test as a signal of changed behavior rather than an immediate bug.

Q: Can characterization tests replace other types of testing? A: No, characterization tests are a complementary practice. They excel at protecting existing, often undocumented, behavior, making them invaluable for legacy systems, refactoring, and upgrades. However, they don't replace the need for unit tests (which validate small, isolated components), integration tests (which verify interactions between components), or end-to-end tests (which validate full user flows) that explicitly test new features and desired outcomes against specifications.