Demystifying Threat Models: A Practical Guide for Developers

As software developers, we frequently encounter the term "threat model," often used as a buzzword rather than a deeply understood concept. From discussions around post-quantum cryptography to the security implications of age verification laws, it's clear that many lack an intuitive grasp of what a threat model truly entails.

While formal threat modeling is a specialized cybersecurity process, you don't need to be an infosec expert to incorporate informal threat modeling into your design and architecture phases. Doing so can significantly improve the security and resilience of your products and services.

The Core Questions of Threat Modeling

At its heart, an informal threat model should answer a few fundamental questions. Think of it as a structured way to anticipate and mitigate potential harm to your system:

1. What are we protecting? Identify your assets – the valuable components or data within your system. Without clarity here, any security effort is unfocused.
2. Who or what wants to harm these assets? Consider various adversaries: malicious actors (hackers, stalkers), environmental factors (natural disasters), disgruntled employees, or even poorly conceived legislation.
3. How might these adversaries attack our assets? Brainstorm specific attack scenarios, remembering Murphy's Law – if something can go wrong, it will.
4. What actions will we take to prevent these attacks? Outline your mitigations and controls. Again, Murphy's Law applies; plan for failures.

While these four questions provide a starting point, a truly effective threat model expands further:

5. How are the assets related or connected? Visualize your system as a graph, not just a list of items. Understanding interdependencies reveals attack paths and critical choke points.
6. What assumptions are we making? This is crucial. Every design decision carries assumptions about the environment, user behavior, or underlying technologies. If an assumption proves false, your entire security posture could be compromised. Being explicit about assumptions helps uncover "unknown unknowns" – like how the "Invisible Salamanders attack" exploited an assumption about key uniqueness in AEAD schemes.
7. What threats are we deliberately not addressing? You cannot secure against every conceivable future attack. Documenting accepted risks and scope limitations is vital to managing expectations and focusing resources effectively.

Remember, threat models are not static documents. They should evolve with your system, adapting as new threats emerge or as your understanding of the system deepens.

A Practical Approach to Getting Started

To build your threat modeling intuition, consider this iterative process:

1. Prepare: Keep the seven questions readily available for quick reference.
2. Map Your System: Visually chart your system's components and their direct interactions. Think of a data flow diagram or a dependency graph.
3. Iterative Deep Dive: Start at a high level, encompassing the entire system. Then, progressively narrow your focus, like a shrinking game zone, to individual components. For each component and interaction, identify its inputs and outputs, and answer as many of the seven core questions as possible.
4. Uncover Assumptions: As you drill down, note any assumptions you're making about the layers you aren't currently scrutinizing.
5. Refine: Look for and eliminate inappropriate or unnecessary relationships between components that could introduce vulnerabilities.

By systematically working from the macro to the micro, you gain a comprehensive understanding of your system's attack surface.

Learning from Examples: The Good and The 'Attempt'

Let's consider two examples to illustrate the spectrum of threat modeling quality.

One positive example is the ongoing effort to deliver key transparency to the Fediverse. Its public specification includes a detailed threat model covering declared assumptions, assets, actors, and categorized risks (e.g., "Prevented by design," "Mitigated," "Addressable," "Open"). While not flawless, it offers a structured and transparent approach.

On the other end, consider Matrix's threat model (v1.18). It's essentially a list of attack types, such as Denial of Service or Spoofing. Critically, it omits:

• Explicit assumptions.
• Identified assets or their relationships.
• Any mention of cryptography or key management, despite Matrix being an end-to-end encrypted messaging system. This omission is particularly glaring given several public cryptographic vulnerability disclosures.
• It has also remained largely unchanged since its v1.1 release in 2021, despite new attacks being discovered.

While Matrix's effort earned a C- and an "an attempt" sticker, it's noted that even a poor threat model is better than none at all – a point where Signal's approach of providing technical specifications without a formal threat model for the user to decipher falls short.

How Threat Models Drive Better Design

Threat models empower defenders to shape the battleground, forcing attackers into predictable corners. They are the foundation of effective defense-in-depth.

Practical Example: Preventing Credential Stuffing

Consider the pervasive issue of credential stuffing, an attack stemming from users reusing passwords. A threat model would identify password reuse as a critical vulnerability. The evolution of solutions highlights threat modeling in action:

• Problem: Humans are bad at remembering many unique, high-entropy passwords.
• Mitigation (Early): Password managers.
• Mitigation (Better): Passkeys. These use asymmetric cryptography for authentication, often backed by hardware security tokens, making them unphishable. This approach removes the burden of memorization from users.

To effectively counter credential stuffing, a threat model would guide you to:

1. Design your application to mandate passkeys.
2. Require users to enroll multiple passkeys for backup and resilience.
3. Implement a secure, auditable "break glass" procedure for administrators to add new passkeys in emergencies.
4. Eliminate password-based authentication entirely where possible (passwords for key derivation can be a different matter).

By understanding the threat and its root cause (human fallibility), you can implement a solution that not only enhances security but also significantly improves usability – a core tenet that "security at the expense of usability comes at the expense of security."

Advanced Example: Distributed End-to-End Encryption

In decentralized E2EE systems, such as those proposed for ActivityPub (Fediverse) or ATProto (BlueSky), threat models reveal complex challenges with protocols like MLS (Message Layer Security). For instance, MLS relies on an "Authentication Service" and precise message ordering for its ratcheting trees.

For ActivityPub, issues like designing a key transparency mechanism for the Authentication Service and handling race conditions from multiple KeyUpdate messages require careful consideration within the threat model. ATProto presents an even trickier scenario due to its lack of instances and global, blockchain-like state, complicating consistent message ordering across many writers, which is critical for MLS's security guarantees.

Understanding these intricate dependencies and potential failure modes through threat modeling is essential to designing robust, decentralized E2EE solutions.

FAQ

Q: What's the main difference between an informal and formal threat model?

A: An informal threat model focuses on building an intuition for potential risks and mitigations, often used during early design phases by developers. A formal threat model is typically a more rigorous, documented, and often academic process, sometimes involving specific methodologies like STRIDE, and is generally conducted by specialized cybersecurity professionals.

Q: Why is it crucial to list assumptions in a threat model?

A: Listing assumptions is crucial because if an underlying assumption about your system, its environment, or user behavior is incorrect, the entire threat model and its proposed mitigations could become invalid. Explicitly stating assumptions helps identify hidden vulnerabilities and allows for a clearer evaluation of accepted risks.

Q: How do threat models help improve usability, not just security?

A: Threat models can highlight scenarios where security measures impede user experience, prompting designers to find solutions that address both. For example, identifying the threat of password reuse leads to solutions like passkeys, which offer stronger security while significantly simplifying the user authentication process by removing the need for memorizing complex secrets.