Where the judge sits matters.
The evaluator cannot be the agent owner, and that's enforced in code, not policy. Inside your VPC, inside your call path, or inside your team's audit tooling all sound independent. None of them are. Here's the distinction.
Blocked at the schema
A submission where submitted_by equals agent_owner is a hard rejection before any audit logic runs, not a downgraded or logged-and-allowed request.
Org-level circumvention closed
Same-organisation submissions are blocked even when the two accounts have different names: subsidiaries and related parties can't route around the guard.
Hardened three times
A Firebase account-type guard, a case/whitespace bypass fix, and a tenant-scoping fix found during a dedicated red-team sweep.
Independent of the customer, not just isolated from other customers
"Independent" gets used loosely in this market. The distinction that matters: whether the judge is organisationally independent of the party whose agent is being evaluated, or merely placed somewhere convenient for that party.
| Approach | Where the judge runs | Independent of the agent owner? |
|---|---|---|
| In-VPC judges (e.g. Fiddler) | Inside the customer's own cloud environment | No, the customer controls the infrastructure the judge runs on, which is placement, not organisational independence. |
| Inline audit (e.g. agent-authority tooling) | In the agent's own call path | No, audit happens as part of the same operational flow the agent owner controls. |
| Owner-operated audit (most eval tooling) | Run by the agent owner's own team | No, the agent owner is marking its own homework, whatever the tool. |
| Jiminy | Independent evaluator organisation, enforced at the schema level | Yes, structurally, not by configuration choice. |
Full technical detail: the independence guard
Structural, not policy
The independence guarantee is enforced as a Pydantic schema validator, a hard ValueError on violation. It cannot be relaxed by a configuration flag or a support override; changing it requires a code change.
Two-account-type model
Firebase Auth backs two structurally distinct account types: Agent Owner (submits traces of agents they operate; can view only their own audit history; cannot initiate audits of their own agents) and Independent Evaluator (receives Agent Owner traces; produces independent verdicts). A Firebase account-type guard enforces this before any audit logic runs for Firebase-authenticated accounts, in addition to the schema-level submitted_by != agent_owner check.
Org-level check
If submitted_by and agent_owner share the same org_id (configured via allow-lists in GCP Secret Manager), the submission is blocked with HTTP 403. String equality on the account name alone is not sufficient: it would miss subsidiaries and related parties, so the org-level check exists specifically to close that gap.
Known gap, stated plainly
The Firebase Auth two-account Firestore model is still maturing: today's guard is string equality on submitted_by != agent_owner plus an org_id check, not yet a fully Firestore-account-type-architectural enforcement. This is tracked as an open item, not hidden.