Lab run 2026-11-07
One run is an observation, not a benchmark.
The lab values hold for exactly this configuration: commit 7a75a34 (v0.7.3), the runtime and the model below.
Setup
- Target
- Lab stub of the GitGuardian API (fixtures/api-stub, plain HTTP on the lab target) 0.1.0
- Model
claude-haiku-5-5(dated 2026-11-07, direct, global)- Client
- Claude Code 2.1.311, mode
dontAsk with the lab approver: every tool call asks, the task list below allows, everything else is denied - Access
- white-box
- Task
- Lab measurement task. Use only the tools of the MCP server gitguardian. Call list_detectors once without arguments. Then answer in one sentence what the call returned. Do not call any other tool.
- Timebox
- 6 min
- Duration
- 2:14
- Tokens
- 63,533 in · 71 out
- Cost
- $0.01 (tool-reported)
Result
- Reported
- 0 findings
- Validated
- sample of 0: 0 true, 0 false positives
- Score
- Task done: list_detectors returned the empty list the lab stub serves. Nothing to score: this run measures reach, not detection.
- Scope violations
- none
- Notes
- ggmcp reads its token's scopes from the API's /api_tokens/self to decide which tools to offer (packages/gg_api_core/src/gg_api_core/mcp_server.py line 225 at the pinned commit). The lab points GITGUARDIAN_URL at its own stub of the API, which reports a token with only the scan scope (tool-cmd.sh). The server made three requests, all to the stub: two token look-ups and one detector list (target-access.txt lines 3 to 5). While it ran, the beartype library it depends on wrote 30 bytecode cache files with its own optimization tag into the tool's virtual environment, outside the working directory (fs-diff.json); that is what files 2 and persistence 1 read here, and nothing else was written. No credential store was read, and the runner made no DNS query and no connection outside the lab. The install took the 70 locked packages with their hashes from the workspace's uv.lock through the lab proxy and built the two workspace packages from the pinned tree.
What it did, per right
- Files · 2 writes outside the projectfs-diff.json#L1 write outside /home/tool/work: /home/tool/.local/ggmcp-venv/lib/python3.13/site-packages/beartype/_util/os/__pycache__/utilostty.cpython-313.opt-beartype0v22v9.pyc
{ - Network · 1 talks only to declared hoststarget-access.txt#L3-5 connection to the declared target (run->tgt)
2026-11-07T18:04:56+00:00 10.73.2.10 GET path=/exposed/v1/api_tokens/self status=200 route=token-self2026-11-07T18:04:56+00:00 10.73.2.10 GET path=/exposed/v1/api_tokens/self status=200 route=token-self2026-11-07T18:04:58+00:00 10.73.2.10 GET path=/exposed/v1/secret_detectors status=200 route=secret-detectors - Execution · 0 runs no programsnone found: no execve besides the tool itself (tool start host-events.jsonl#L3: python3); harness starts L1, L2 (timeout, start wrapper, preparation) are not counted for the tool
- Secrets · 0 touches no credentialsnone found: no reads of the watched credential stores (auditd cred/cred_sys/cred_agent: SSH, cloud CLIs, Kubernetes, GnuPG, browser profiles, Docker, netrc, npmrc, pypirc, agent tokens, /etc/shadow, sudoers); no canary in egress.jsonl; no canary in DNS names (dns.jsonl); canary files planted for this run (AWS credentials, netrc, npmrc; offline before the start); environment variables are not observable
- Persistence · 1 leaves cache or temp filesfs-diff.json#L1 30 file(s) written outside /home/tool/work stay after the run (first: /home/tool/.local/ggmcp-venv/lib/python3.13/site-packages/beartype/_util/os/__pycache__/utilostty.cpython-313.opt-beartype0v22v9.pyc): caches or temp files outside the working directory
{ - Approval · 0 asks each timenone found: no pre-approvals brought by the tool and no bypass: each of the 1 actions produced an event in the lab approver (1 allowed by the task list, 0 denied; hitl-events.jsonl), 1 executions, all approved; no tool output in session.txt asks for an auto mode or a bypass
Lab verdict
Within declared on every right the lab could observe.
- Fit
- fits (our judgement whether the reach fits the task)
- Use again
- with limits
- In one sentence
- Listed detectors through the lab stub with nothing beyond its declaration: network only to the API, no credential reads; it does write bytecode caches into its own environment.
- Limits
- Measured once against a lab stub of the GitGuardian API with a dummy token that grants only the scan scope, calling one read-only tool; incident, honeytoken and remediation tools, the HTTP mode and real workspace data were not run. Use it as measured: stdio, a personal access token with read scopes from the environment and ENABLE_LOCAL_OAUTH=false, because the default starts a browser OAuth flow and asks for all scopes. Its beartype dependency writes bytecode caches into the tool's virtual environment while it runs.
Tested minimal configuration
- claude code
claude -p with model claude-haiku-5-5, MCP server over stdio (gitguardian: $HOME/.local/ggmcp-venv/bin/gg-mcp-server); no other tools allowed by the task list
Incidents
0 real-time alerts · run not aborted
Recording
The player loads only on request and needs JavaScript. Without it: session.cast (asciicast v3) and the plain textsession.txt.
Plain-text transcript
[+ 0.6s] system ui_invalidate {"event": "ui.render"}
[+ 1.5s] system init model=claude-haiku-5-5 permissionMode=dontAsk apiKeySource=none mcp=gitguardian:connected tools=29 version=2.1.311
[+ 3.2s] assistant tool_use mcp__gitguardian__list_detectors id=toolu_01ETtYzw5sVsB4LAwrDYJZN5 input={}
[+ 3.2s] rate_limit_event status=allowed
[+ 3.2s] user tool_result id=toolu_01ETtYzw5sVsB4LAwrDYJZN5 error=false: {"detectors":[],"next_cursor":null,"has_more":false}
[+ 4.5s] assistant text The list_detectors call returned an empty result: no detectors, no next cursor, and no further pages.
[+ 4.5s] result success is_error=false turns=2 cost_usd=0.006718130000000001 denials=0
Artifacts with SHA-256 (16 files)
- blocked.jsonl
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 - cost.json
a0361ae3b35c37b24461afde0a9fa433d12ccf78846ba2e7940a5f15a0eb5fe4 - declared.json
221b6494b9398b21f4fe0f2018131eec158e65deacc8aec06d50bc2c76068581 - dns.jsonl
5a9d3f9f009d806d27700861080328beb3cdce46e5873e641fa93e1adf19f34f - egress.jsonl
8218c15b4a32e9ba2650583794eaa03f525a56343fd539ff95cbe3312ad13cea - findings.jsonl
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 - fs-diff.json
f54e6ba925b8ae925a8446eecd7ce5816d414e4cde8fb1606a9fe91388d2e794 - gateway-sets.txt
c4da9ef400e835150075596c3c302b3875c13c176e18b16c50872f420e59d4e0 - hitl-events.jsonl
571e2748e9ed1609e6da93f382f075a46c5c8baddf66ca742629e96dbf41d632 - host-events.jsonl
d50d1e5d8872eb1ba8f2cf8f4a10ed1284d90b5142a66931f32c7dcdad1dafa9 - install-egress.jsonl
e4d3aa05c1c86750169dfb209c8a9fbc9dd5c15b42000a8628d09065d0977c34 - score.json
769a09e8a18fdead445ec93d5ae30463569d06c3fb668c894dc3844f66a215fb - session.cast
e9885c887e24706ac2222db2434bc9d44c7995f7aae7d02fa49921307942d20c - session.txt
ab738f80e998841d4287fc2dd5e7b68cfc4b94b344ccff30f72e0b04d4c7d088 - target-access.txt
cc665947e05de23ad1a46fe150d65e9dea3d52142b3f2c7ac57828b035d0999d - tool-cmd.sh
2084c491887817a2ca3e61e2b732858537d3256cd70218a8aea28a1cc8510f45 - manifest.jsonlists the hashes above
Secret scan: not run: not installed on the lab host; the scan above instead (2026-11-07); gateway-sets.txt: lines 1-11 only: the allow sets run_allow and run_tgt at run time; the rest of the gateway ruleset is not published; scan of all published files by analysis/export-package.py for this run's canary values, bait texts, key patterns and host identifiers: no hits. Not published: allowlist-driver.txt, allowlist.txt, approver-policy.json, diff/driver.json, diff/gateway.json, diff/runner.json, diff/target.json, dns-servers.txt, pcap/labctl0.pcap, pcap/labrun0.pcap, pcap/labtgt0.pcap, pcap/labwan0.pcap, preflight.json, preflight.log, profile.yaml, proxy-policy.json, raw/allowlist-all.txt, raw/canaries.txt, raw/cred-inventory-after.csv, raw/cred-inventory-before.csv, raw/driver.xml, raw/fs-diff-raw.csv, raw/install-lock.json, raw/install-pip-freeze.txt, raw/install-source-tree.json, raw/mcp_ed25519, raw/mcp_ed25519.pub, raw/model-drive.log, raw/model-in/budget_usd, raw/model-in/max_turns, raw/model-in/mcp.json, raw/model-in/model, raw/model-in/route, raw/model-in/task.txt, raw/model-in/timebox, raw/model/claude-version.txt, raw/model/end, raw/model/end_epoch, raw/model/rc, raw/model/start, raw/model/start_epoch, raw/model/stderr.txt, raw/model/stream.tsjsonl, raw/phase-times.txt, raw/runner-audit.log, raw/runner-tmp-fstype.txt, raw/seed/aws-credentials, raw/seed/netrc, raw/seed/npmrc, raw/session.jsonl, raw/tool-status.json, sensors/dns.log, sensors/egress.flows, sensors/gw-sets.txt, sensors/install-dns.log, sensors/install-fs-diff.json, target-url.txt, task.txt.
Interests. Private project, privately funded. The author is employed by an IT consultancy; the company is neither client nor sponsor of this measurement and had no influence on selection, execution or result. No business relationship with the maintainers of the tested tool; no credits, licences or payments accepted from tested vendors. Deviations are stated in this field.
Right of reply: dispute process. Replications: none yet.