What you will learn

Map assets before prompts
Trace an abuse case to a control
Make the first measurement honest

Before you start

Tool Calling

The flow

1.Identify scope
2.Inspect evidence
3.Apply host control
4.Run negative cases
5.Compare benign behavior
6.Review remaining risks

Architecture and boundaries

Caller → host

Authenticated identity selects scope; untrusted input cannot grant permission.

Retrieval → model

Only current, approved, authorized evidence crosses the boundary.

Model → tool

The host validates a proposal, budget and approval before an effect.

Output → user

Structured facts and source checks are distinct from natural-language truth.

Map assets before prompts

A RAG answer can expose documents; an agent can change resources. Start with the asset and the person authorized to use it. Draw caller → host → retrieval → model → tool/UI and mark every boundary. The model is a probabilistic proposal generator, not an identity or permission authority.

Record who controls user text, document content, tool descriptions and approval records. In the lab, tenant/user/run identity comes from the trusted fixture host. A document cannot choose that scope. The threat register names an owner and a negative test for each crossing.

Trace an abuse case to a control

An imported document may ask the model to export evidence. Preventing that effect requires a narrow tool surface and host authorization, not just a sentence in the system prompt. Keep sensitive data out of the context whenever the task does not need it.

Use threat-model.json to connect T-01 document redirection, T-02 false claims, T-03 grant replay and T-04 budget exhaustion with controls. Distinguish a control you tested from a production dependency such as authentication, network enforcement or durable storage.

Make the first measurement honest

The first baseline is a deterministic policy regression, not an attack success estimate for a live model. Author expectations and independent human ground truth are different. Track benign successes as well as denials so a host that refuses everything is not celebrated.

Before release, assign a risk owner, review the exact candidate and document unresolved boundaries. The lab cannot prove an OS sandbox or deploy an authorization service; its report keeps human review and rollback readiness pending.

Try it

threat-model.json

json

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

{
  "version": "2026-10-31.1",
  "owner": "Interview Preparator teaching lab",
  "assets": ["tenant documents", "approval grants", "run state", "source provenance", "resource budgets"],
  "actors": ["authenticated user", "untrusted document author", "probabilistic model", "trusted host", "authorization service"],
  "boundaries": [
    {"from":"caller", "to":"host", "control":"authenticated identity; request validation"},
    {"from":"retrieval", "to":"model", "control":"current tenant/user ACL; approved, live sources"},
    {"from":"model", "to":"tools", "control":"closed tool schema; host permission and payload-bound grant"},
    {"from":"model", "to":"UI", "control":"escaped plain text; source/claim contract"},
    {"from":"host", "to":"outside world", "control":"no external side-effect tool in this fixture; production egress enforcement is separate"}
  ],
  "risks": [
    {"id":"T-01", "threat":"document instruction redirects tool use", "owner":"host policy", "test":"injection"},
    {"id":"T-02", "threat":"valid source ID paired with false claim", "owner":"reliability evaluator", "test":"contradicted"},
    {"id":"T-03", "threat":"replayed or scope-changed write approval", "owner":"approval adapter", "test":"grant"},
    {"id":"T-04", "threat":"budget exhaustion through repeat calls", "owner":"runtime", "test":"budget"}
  ]
}

Run npm run examples:security and npm run eval:security. This source is a deterministic teaching fixture; it does not prove live model safety, NLI, production auth or OS/network isolation.

Interview questions

Who grants a model permission?

Record who controls user text, document content, tool descriptions and approval records. In the lab, tenant/user/run identity comes from the trusted fixture host. A document cannot choose that scope. The threat register names an owner and a negative test for each crossing.

Which boundary contains document-driven tool misuse?

Use threat-model.json to connect T-01 document redirection, T-02 false claims, T-03 grant replay and T-04 budget exhaustion with controls. Distinguish a control you tested from a production dependency such as authentication, network enforcement or durable storage.

What does the fixture baseline fail to establish?

Before release, assign a risk owner, review the exact candidate and document unresolved boundaries. The lab cannot prove an OS sandbox or deploy an authorization service; its report keeps human review and rollback readiness pending.

Practice

Extend one negative scenario and its authorized positive counterexample for AI Threat Modeling & Trust Boundaries.

Check your solution

State the trusted input and effect boundary.
Assert the expected status and observed effect.
Report coverage and limitations without claiming production readiness.

AI Threat Modeling & Trust Boundaries