Two Secret Scanners Saw Different Leaks

Some disagreement between secret scanners seems reasonable to me. Different rules, different parsers, different decisions about whether a finding is worth showing.

The overlap in a recent Gitee study was small enough to make me stop and think about what a scanner report actually tells us.

Ben Zimmermann reported scanning 7,256,531 public repositories with TruffleHog and Kingfisher, finding 26,400 unique credentials verified live at scan time. TruffleHog accounted for 12,737 findings, Kingfisher for 14,664, and 1,001 appeared in both sets. The findings were deduplicated by secret value. Read the research.

That does not tell me that every scanner misses half the secrets in every environment. It does make me want a better answer than “we have secret scanning turned on.”

The combined result is not the total exposure

Both scanners could miss the same credential. A verified-only result also leaves out candidates that could not be confirmed. The combined findings are a useful observed set, not a complete inventory of every secret that existed in those repositories.

I would read the result as evidence of complementary coverage in that run, not as a universal recall measurement or a permanent product ranking.

There is publication context to keep in view, too. The research was developed with MongoDB’s Kingfisher team, and Modal sponsored compute. Revocation outcomes were unconfirmed when the post was published. That does not invalidate the work; it does make it important to keep the reported experiment separate from a purchasing recommendation. Methodology and disclosure notes.

The provider list interested me more than a league table

The study reports provider-specific differences, including AI services that one tool covered better than the other. Those observations belong to the tested rules and versions, rather than a timeless statement about what either product supports.

For an agent-security program, I would start with the services the agents and their developers can actually use. An approved-provider list is one input. Dependencies, gateways, development tools and exceptions may introduce others.

For each credential type, the review questions are fairly specific:

• Is there a detector for it in the scanner we deploy?
• Which version and ruleset did we check?
• Does the scan include Git history, or only the current working tree?
• Can the result distinguish a candidate from a credential that was actively verified?
• What happens when verification is disabled, unavailable or inconclusive?

That is a more useful conversation than comparing the headline number of detectors in two products. A large rule library can still leave a particular integration unexamined.

A quiet report can have several explanations

No findings might mean that no recognizable secrets were present. It might also mean that the source was not scanned, its history was omitted, the relevant detector was absent, or a reporting filter hid the result.

Those are different outcomes. A single green status would leave me guessing which one occurred.

TruffleHog’s documentation distinguishes verified, unverified and unknown verification outcomes. Network or API errors are among the reasons verification can be unknown. That distinction is useful: a failed verification attempt is not the same as proof that the credential is harmless. TruffleHog verification documentation.

My preferred coverage record would say what ran and what it could establish. For example: the repository and refs examined, scanner version, ruleset, exclusions, verification setting and any collection errors.

An internal identifier or a suitable protected fingerprint can help correlate findings. Raw secrets do not belong in the coverage record, a ticket title or a chat discussion.

Compare tools against a fixture you control

For my own coverage review, a small private fixture would be a more useful starting point than repeating a scan of millions of public repositories.

The proposed exercise is deliberately modest:

1. Pick credential types used by a few real integrations. Record the owner and intended scope, without copying production secrets.
2. Build a private fixture with synthetic examples for pattern detection. Include a harmless historical commit and a current-tree example so history coverage can be checked separately.
3. For verification testing, use only owner-authorized, short-lived test credentials in an isolated fixture, with tightly limited authority and usage. Never publish them or rely on them being free to use.
4. Run pinned scanner versions against the same repository snapshot and refs. Record exclusions, verification behavior and failures.
5. Compare finding identities and source locations, not just totals. Investigate why a particular fixture was absent from one result.
6. Revoke any test access, confirm that it stopped working, and remove sensitive test artifacts according to the test plan.

A synthetic token may be useful for testing pattern recognition without ever being valid. It is not evidence that live validation works. Conversely, a validation test does not prove that every source format or historical ref is being scanned.

Kingfisher explicitly notes that live validation and blast-radius operations make provider API requests and require authorization. I would review those behaviors before running a comparison, not discover them from the outbound traffic afterward. Kingfisher documentation.

This is a proposed fixture exercise. I have not run a scanner benchmark or tested third-party credentials for this article.

A coverage sheet small enough to maintain

I have included a blank AI credential-scanning review template. It is a short Markdown document to copy for each provider and credential type, with results initially marked Not assessed. It can live beside a repository’s security notes without requiring a spreadsheet or a new dashboard.

Each record keeps five questions visible:

Record	What it should answer
Provider and credential class	Which kind of access are we trying to find?
Scanner and ruleset	What exact implementation did we assess?
Source and history scope	Where did we look, and what did we exclude?
Fixture and verification outcome	What did our authorized test establish?
Owner and revocation evidence	Who responds, and how do we confirm access ended?

The template leaves room for the supporting references. A second scanner is useful when it closes a gap that matters to the environment. That is something the fixture can help establish.

A completed record needs a review date. An assessment from six months ago may need another look after a provider changes its credential format, a scanner changes its rules, or the organization adds a new source-hosting service.

Finding a key is the beginning of the response

A plausible exposed secret should not have to pass an authentication test before anyone takes it seriously. I would identify its owner, assess the exposure and follow the appropriate containment process while preserving the evidence needed for investigation.

Removing a value from the latest commit is also different from revoking its access. OWASP’s secrets-management guidance treats revocation and rotation as explicit response work. OWASP Secrets Management Cheat Sheet.

For program reporting, I would prefer a few operational measures over an impressive scan count: coverage of the credential types we use, unresolved collection errors, time to owner assignment, and time to confirmed revocation. Each measure needs a defined clock. The time a secret was committed is not necessarily the time it was issued, first exposed elsewhere or last used.

The Gitee research is a good reason to ask where else our code and integrations live. It is not a reason to assume that every organization needs the same global monitoring footprint.

In An AI Key Leaked. Would We Notice?, I look at the provider activity after a credential leaves its intended setting. Here I would start with one provider we use and follow a known test fixture through the scanner’s result. If the result is quiet, I would like to know why.