Lab run 2026-11-07
One run is an observation, not a benchmark.
The lab values hold for exactly this configuration: commit 5d147dc (v4.8.6), the runtime below.
Setup
- Target
- Lab fixture vuln-mcp-server (MCP over streamable HTTP) 0.1.0
- Model
- none (the tool ran without a model)
- Client
- Standalone (own model access) n/a (no agent client), mode
n/a (no approval layer) - Access
- white-box
- Task
- Run without a model: install the Cisco mcp-scanner CLI, start it in YARA-only mode against the prepared MCP server and observe the scanner's reach and any egress. Expected: a connection to the target only, no egress to the internet.
- Timebox
- 10 min
- Duration
- 2:38
Result
- Reported
- 3 findings
- Validated
- sample of 3: 3 true, 0 false positives
- Score
- At tool level all 3 baited tools were flagged HIGH and both clean controls stayed SAFE. The prompt and server-instruction probes were not scanned: the remote scan lists tools only.
- Scope violations
- none
- Notes
- A small synthetic sample from a lab fixture, so no general detection rate follows: coverage 3 of 5 probes, recall 1.0 on the scanned ones (score.json). Even in YARA-only mode without keys the scanner tries at start-up to download litellm's model cost map from GitHub (tool-stderr.txt line 2). The lab resolver answered that name with its sinkhole (dns.jsonl line 1), the connection attempts were dropped (blocked.jsonl), and the scan began about six seconds after the tool started: tool start 13:02:35 (host-events.jsonl line 4), first request 13:02:41 (target-access.txt line 2). During the install phase the same host was requested once through the lab proxy, one second after the last package download, and refused (install-egress.jsonl line 202). The install took the 99 locked packages with their hashes from the project's uv.lock.
What it did, per right
- Files · ?no evidence: no writes besides result paths (fs-diff.json); reads are not audited, so 0 (reads only the working directory) and 1 (reads outside) cannot be told apart; /tmp in the runner is a tmpfs and invisible to fs-diff
- Network · 2 talks to any host
beyond declaredblocked.jsonl#L1-5 the runner (10.73.2.10) tried five times, a second apart, to connect to 198.18.0.1:443, the lab's DNS sinkhole, and was dropped (line 1 lies half a second before the tool start the host recorded, because the lab machines' clocks differ by about a second){"timestamp": "2026-11-07T13:02:34.537212", "dvc": "Netfilter", "raw.pktlen": 60, "raw.pktcount": 1, "oob.prefix": "gw-drop-fwd ", "oob.time.sec": 1794056554, "oob.time.usec": 537212, "oob.mark": 0, "oob.ifindex_in": 4, "oob.ifindex_out": 2, "oob.hook": 2, "raw.mac_len": 14, "oob.family": 2, "oob.protocol": 2048, "raw.label": 0, "raw.type": 1, "raw.mac.addrlen": 6, "ip.protocol": 6, "ip.tos": 0, "ip.ttl": 63, "ip.totlen": 60, "ip.ihl": 5, "ip.csum": 55634, "ip.id": 36867, "ip.fragoff": 16384, "src_port": 45344, "dest_port": 443, "tcp.seq": 2585627822, "tcp.ackseq": 0, "tcp.window": 64240, "tcp.offset": 0, "tcp.reserved": 0, "tcp.urg": 0, "tcp.ack": 0, "tcp.psh": 0, "tcp.rst": 0, "tcp.syn": 1, "tcp.fin": 0, "tcp.res1": 0, "tcp.res2": 0, "tcp.csum": 53908, "src_ip": "10.73.2.10", "dest_ip": "198.18.0.1"}{"timestamp": "2026-11-07T13:02:35.555233", "dvc": "Netfilter", "raw.pktlen": 60, "raw.pktcount": 1, "oob.prefix": "gw-drop-fwd ", "oob.time.sec": 1794056555, "oob.time.usec": 555233, "oob.mark": 0, "oob.ifindex_in": 4, "oob.ifindex_out": 2, "oob.hook": 2, "raw.mac_len": 14, "oob.family": 2, "oob.protocol": 2048, "raw.label": 0, "raw.type": 1, "raw.mac.addrlen": 6, "ip.protocol": 6, "ip.tos": 0, "ip.ttl": 63, "ip.totlen": 60, "ip.ihl": 5, "ip.csum": 55633, "ip.id": 36868, "ip.fragoff": 16384, "src_port": 45344, "dest_port": 443, "tcp.seq": 2585627822, "tcp.ackseq": 0, "tcp.window": 64240, "tcp.offset": 0, "tcp.reserved": 0, "tcp.urg": 0, "tcp.ack": 0, "tcp.psh": 0, "tcp.rst": 0, "tcp.syn": 1, "tcp.fin": 0, "tcp.res1": 0, "tcp.res2": 0, "tcp.csum": 53908, "src_ip": "10.73.2.10", "dest_ip": "198.18.0.1"}{"timestamp": "2026-11-07T13:02:36.579310", "dvc": "Netfilter", "raw.pktlen": 60, "raw.pktcount": 1, "oob.prefix": "gw-drop-fwd ", "oob.time.sec": 1794056556, "oob.time.usec": 579310, "oob.mark": 0, "oob.ifindex_in": 4, "oob.ifindex_out": 2, "oob.hook": 2, "raw.mac_len": 14, "oob.family": 2, "oob.protocol": 2048, "raw.label": 0, "raw.type": 1, "raw.mac.addrlen": 6, "ip.protocol": 6, "ip.tos": 0, "ip.ttl": 63, "ip.totlen": 60, "ip.ihl": 5, "ip.csum": 55632, "ip.id": 36869, "ip.fragoff": 16384, "src_port": 45344, "dest_port": 443, "tcp.seq": 2585627822, "tcp.ackseq": 0, "tcp.window": 64240, "tcp.offset": 0, "tcp.reserved": 0, "tcp.urg": 0, "tcp.ack": 0, "tcp.psh": 0, "tcp.rst": 0, "tcp.syn": 1, "tcp.fin": 0, "tcp.res1": 0, "tcp.res2": 0, "tcp.csum": 53908, "src_ip": "10.73.2.10", "dest_ip": "198.18.0.1"}{"timestamp": "2026-11-07T13:02:37.603228", "dvc": "Netfilter", "raw.pktlen": 60, "raw.pktcount": 1, "oob.prefix": "gw-drop-fwd ", "oob.time.sec": 1794056557, "oob.time.usec": 603228, "oob.mark": 0, "oob.ifindex_in": 4, "oob.ifindex_out": 2, "oob.hook": 2, "raw.mac_len": 14, "oob.family": 2, "oob.protocol": 2048, "raw.label": 0, "raw.type": 1, "raw.mac.addrlen": 6, "ip.protocol": 6, "ip.tos": 0, "ip.ttl": 63, "ip.totlen": 60, "ip.ihl": 5, "ip.csum": 55631, "ip.id": 36870, "ip.fragoff": 16384, "src_port": 45344, "dest_port": 443, "tcp.seq": 2585627822, "tcp.ackseq": 0, "tcp.window": 64240, "tcp.offset": 0, "tcp.reserved": 0, "tcp.urg": 0, "tcp.ack": 0, "tcp.psh": 0, "tcp.rst": 0, "tcp.syn": 1, "tcp.fin": 0, "tcp.res1": 0, "tcp.res2": 0, "tcp.csum": 53908, "src_ip": "10.73.2.10", "dest_ip": "198.18.0.1"}{"timestamp": "2026-11-07T13:02:38.627275", "dvc": "Netfilter", "raw.pktlen": 60, "raw.pktcount": 1, "oob.prefix": "gw-drop-fwd ", "oob.time.sec": 1794056558, "oob.time.usec": 627275, "oob.mark": 0, "oob.ifindex_in": 4, "oob.ifindex_out": 2, "oob.hook": 2, "raw.mac_len": 14, "oob.family": 2, "oob.protocol": 2048, "raw.label": 0, "raw.type": 1, "raw.mac.addrlen": 6, "ip.protocol": 6, "ip.tos": 0, "ip.ttl": 63, "ip.totlen": 60, "ip.ihl": 5, "ip.csum": 55630, "ip.id": 36871, "ip.fragoff": 16384, "src_port": 45344, "dest_port": 443, "tcp.seq": 2585627822, "tcp.ackseq": 0, "tcp.window": 64240, "tcp.offset": 0, "tcp.reserved": 0, "tcp.urg": 0, "tcp.ack": 0, "tcp.psh": 0, "tcp.rst": 0, "tcp.syn": 1, "tcp.fin": 0, "tcp.res1": 0, "tcp.res2": 0, "tcp.csum": 53908, "src_ip": "10.73.2.10", "dest_ip": "198.18.0.1"}
dns.jsonl#L1 the runner asked for raw.githubusercontent.com and was given the sinkhole address{"ts": "2026-11-07T13:02:34Z", "client": "10.73.2.10", "port": 46072, "qtype": "A", "name": "raw.githubusercontent.com", "outcome": "sinkhole", "answers": ["198.18.0.1"], "upstream": null, "log_line": 24}
tool-stderr.txt#L2 the scanner logs that its litellm dependency could not fetch the model cost map from raw.githubusercontent.com and fell back to a local copy13:02:41 - LiteLLM:WARNING: get_model_cost_map.py:290 - LiteLLM: Failed to fetch remote model cost map from https://raw.githubusercontent.com/BerriAI/litellm/main/model_prices_and_context_window.json: timed out. Falling back to local backup.
gateway-sets.txt#L2-5 no external target was allowed during the run (run_allow has no elements)set run_allow {type ipv4_addrflags interval}
target-access.txt#L2-4 the scan itself reached only the declared target (run->tgt)2026-11-07T13:02:41+00:00 10.73.2.10 POST methods=initialize2026-11-07T13:02:41+00:00 10.73.2.10 POST methods=notifications/initialized2026-11-07T13:02:41+00:00 10.73.2.10 POST methods=tools/list - Execution · 0 runs no programsnone found: no execve besides the tool itself. In host-events.jsonl lines 1 to 3 are the lab harness (timeout, the start wrapper and a uname the wrapper runs before handing over), line 4 is the tool starting (python3 running mcp-scanner), and nothing was started after it
- Secrets · 0 touches no credentialsnone found: no reads of the watched credential stores (auditd: SSH, cloud CLIs, Kubernetes, GnuPG, browser profiles, Docker, netrc, npmrc, pypirc, agent tokens, /etc/shadow, sudoers), which include the three canary files planted for this run (AWS credentials, netrc, npmrc); no canary in egress.jsonl or dns.jsonl. Environment variables are not observable
- Persistence · 0 leaves results onlynone found: no persistence paths in fs-diff.json (outside the lab's churn filter) and no writes to the watched start-up files (shell rc, autostart, systemd --user)
- Approval · ?no evidence: no approval events, the CLI ran without an agent client or an approval layer
Lab verdict
Beyond declared on 1 of 6 rights: network (observed 2, declared 1), with blocked attempts to reach GitHub at start-up even in YARA-only mode.
- Fit
- partial (our judgement whether the reach fits the task)
- Use again
- with limits
- In one sentence
- Useful as a key-free first pass over an MCP server's tools: it flagged all three baited tools without a false alarm, but it reads tools only and reaches for GitHub at start-up.
- Limits
- Measured only in YARA-only mode against one small synthetic fixture, so no detection rate follows. The default remote scan lists tools only: prompts, resources and server instructions stay unscanned. At start-up the scanner fetches a model price list from GitHub that it does not need for a YARA scan; run it without internet access or behind an allowlist, where the fetch fails and the scan goes on with the bundled copy, as observed. The API and LLM analyzers, the known-configs scan and the stdio launch mode were not run and keep the code reading.
Tested minimal configuration
- cli
mcp-scanner --log-level error --analyzers yara --format raw --server-url <target URL>, without API keys and without cloud upload (tool-cmd.sh)
Incidents
0 real-time alerts · blocked.jsonl lines 1 to 5: SYN packets from the runner to the lab's DNS sinkhole on port 443, all dropped; install-egress.jsonl line 202: a CONNECT to raw.githubusercontent.com during the install phase, refused by the lab proxy (403) · run not aborted
Recording
No recording: a stand-alone tool started by the lab harness has no interactive session. Its call and output are listed with the artifacts below.
Artifacts with SHA-256 (15 files)
- blocked.jsonl
fa4c5b8f74ea463aac65fb397b57cdf0be07dae689dd7e885d292da586bb14fc - declared.json
533c113c0389ad9ef26e7d79419bc8211479489182f60b37cd3c1fb1dbfaa2f1 - dns.jsonl
06554466cecd14c4f098eaa7a3154bd93f085bdf65c47463b3b03c670dcf820f - egress.jsonl
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 - findings.jsonl
38c975b0cb6407806f9a23a4b1d1b24f11467634cc9b6964efa0b3620dbd74f4 - fs-diff.json
10adc8307b205ac94c44019718c7a166b86cb293e41e06f52eec1954af8d8730 - gateway-sets.txt
c4da9ef400e835150075596c3c302b3875c13c176e18b16c50872f420e59d4e0 - hitl-events.jsonl
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 - host-events.jsonl
a76168c86c8c100c9c3afd51bb859cb89cf1087c908213d548bd137d8fed64d5 - install-egress.jsonl
688725a12db77153a69cdccafdbf46a8410b5f4e2290edaadb2ebfb5397aaf2f - scanner-output.json
0812f899fd38cf50b70f21a07fd6dc7a6b87d807cc461bbb844c9f990bca966c - score.json
7412f736197ed83df079596623f9d03df4586a3cd7ffef4f530af8a5ba9af925 - target-access.txt
2dc1685123597a36321643a56e66a7111812b87095df9385b4d8954238c5f2b8 - tool-cmd.sh
0fbcb2b8d1763bfbe48551ac23f31617b44acb8e5eb80c86a2784c9b88030507 - tool-stderr.txt
0de3bb8622c0289de4f199e8fab47ccade21577281c958c78779c90d4cb8dff1 - manifest.jsonlists the hashes above
Secret scan: not run: not installed on the lab host; the scan above instead (2026-11-07); bait texts of the lab fixture in scanner-output.json replaced by their SHA-256, ANSI colour codes and CRLF removed from tool-stderr.txt, the host path removed from score.json, only the two allow sets of the gateway ruleset published (gateway-sets.txt); scan of all published files by analysis/export-package.py for this run's canary values, the original bait texts, AWS keys, private keys, API tokens and host identifiers: no hits. Not published: pcap/, diff/, raw/runner-audit.log, raw/tool-stdout.txt, raw/canaries.txt, raw/seed/, raw/fs-diff-raw.csv, preflight.log, preflight.json, profile.yaml, task.txt, allowlist.txt, approver-policy.json, proxy-policy.json, sensors/gw-nft.txt, sensors/egress.flows, sensors/dns.log, sensors/install-dns.log, sensors/install-fs-diff.json.
Interests. Private project, privately funded. The author is employed by an IT consultancy; the company is neither client nor sponsor of this measurement and had no influence on selection, execution or result. No business relationship with the maintainers of the tested tool; no credits, licences or payments accepted from tested vendors. Deviations are stated in this field.
Right of reply: dispute process. Replications: none yet.