Methodology · v0.1.3 · Updated 2026-09-19
How every grade is computed.
OpenTrustBench grades come from 8 static rules, a permission manifest, and documentary provenance — nothing else. No ML, no black box, no hidden signals. Everything below is reproducible with one command.
The rule suite
Eight rules, two OWASP lists.
Each finding carries a rule ID, a CWE, a remediation, and file:line evidence. Capability checks evaluate de-stringed code, so string literals and prose never confer capabilities.
| Rule | What it detects | Severity | OWASP |
|---|---|---|---|
| AT-SEC-001 | Direct prompt concatenation (injection via template) | Critical | LLM01 |
| AT-SEC-002 | Hardcoded credential or API secret | Critical | LLM02 |
| AT-SEC-003 | Unbounded dynamic shell execution | Critical | ASI02 |
| AT-SEC-004 | eval() / Function constructor invocation | Critical | ASI05 |
| AT-SEC-005 | Unrestricted recursive file deletion | High | ASI02 |
| AT-SEC-006 | Unfiltered SSRF / arbitrary network egress | High | LLM06 |
| AT-SEC-007 | Raw secret leakage in debug logging | Medium | LLM02 |
| AT-COMP-001 | Missing human-in-the-loop gate for critical actions | High | ASI09 |
OWASP references: LLM Top 10 2025 (LLM01–LLM10) and Agentic Top 10 published 2025-12-09 (ASI01–ASI10).
Scoring
Five dimensions, fixed weights.
| Dimension | Weight | What moves it |
|---|---|---|
| Security | 35% | Critical −25, high −15, medium −8, low −3 per finding |
| Permissions | 25% | Excessive −45, broad −25, moderate −10; shell, deletion, unapproved email subtract; enforced human approval adds |
| Provenance | 15% | License, lockfile, security policy, changelog, signatures add documentary points |
| Reliability | 15% | Reliability/compliance findings subtract; human gates add |
| Stability | 10% | Lockfile and changelog presence |
Grades: A 90+ · B 75+ · C 60+ · D 40+ · F below 40 · U ungraded (no score issued). Finding IDs are deterministic hashes, so repeat scans produce byte-stable SARIF. Trust Cards follow the opentrustbench/trust-card/v2 schema (status: graded | ungraded, nullable score and breakdown).
Coverage gating
Insufficient coverage withholds the grade.
Every card carries a coverage object: mode (static-heuristic), status (sufficient or limited), reasons, limitations, and per-file lists (discovered, analyzed, unsupported, excluded, read errors, truncations, limits). Supported code is JavaScript, TypeScript, and Python; JSON/YAML receive only the credential rule. Skill instructions and other prose are not code-analyzed.
A card grades U when coverage is limited: unsupported source present, scan limits reached, or no non-empty supported source analyzed. The CLI exits with code 2 (artifacts still written), and --fail-on treats insufficient coverage as a gate failure, separate from the severity gate. Ungraded cards are excluded from registry rankings and averages, and badges render gray ("U ungraded").
Current snapshot (2026-09-18): 3 of 53 cards graded, 50 ungraded — mostly real-world repos carrying Dockerfiles, type declarations, docs, and images outside rule scope. Withheld, never inflated.
Limits
What the scanner cannot see.
Static analysis only. No code executes, no payloads fire, no runtime behavior is observed. Findings need human triage; counts may include test and example code. Attack analysis is heuristic relabeling of findings; workflow eval parses suites without executing them. Trust Cards are evidence input — never a certification, audit, or compliance verdict.
Why our own repo grades U, not A
It contains its own test ammunition: a deliberately vulnerable fixture, unit tests with attack strings, and automation scripts that shell out by design — plus 66 unsupported implementation files against 60 analyzed, so coverage is limited and the grade is withheld. The shipped CLI grades A. Grade the artifact, not the test bench.
Reproduce any grade
npx @opentrustbench/cli scan https://github.com/owner/repo --quiet echo $? # 0 clean, 1 gate failed, 2 ungraded (limited coverage)
Registry snapshots record the upstream commit sha, so any historical grade can be re-verified exactly.