Four secret scanners compared on four codebases

What every scanner finds#

Most secret scanners find the same easy things. Point any of the four tools in this comparison at a repository and they will report the private keys, the cloud and API keys, and the session tokens. All four did that on every codebase we tested.

The differences appear with the secrets that don't look like secrets. A hashed password is just a long string of letters and numbers. A database password can be an ordinary word with nothing random about it. A credit card number, a crypto wallet recovery phrase, a configuration value that ends up bundled into a web page: none of these match the format of a known token, and none of them stand out as random text. Whether a scanner reports them depends on how it works.

Three of these tools, Gitleaks, TruffleHog, and detect-secrets, scan the raw text. They match it against patterns for known secret formats, and they flag strings that look random enough to be a key. The fourth, Trestle, reads the structure of each file the way the program that loads the file would. It sees each value together with the field it belongs to, so it recognizes a password by that field's name, even when the value looks like an ordinary word. This comparison measures what that difference is worth, using each tool's default settings on public code anyone can download and check.

How the test was run#

The four codebases are public and were chosen to be different from one another. Two are insecure demo projects, one is a set of credentials committed on purpose, and one is a clean software release that should contain no secrets at all. Together they test two separate things: how much real material a scanner finds, and how much unrelated noise it produces when there is little or nothing to find.

Each tool ran with its default settings against the full set of files: no exclusions, no allow-lists, and no saved baseline. That is what a developer gets the first time they run it. The finding counts come straight from each tool's own output, and whether a finding is real, low-value, or a false alarm was decided by checking it against the known contents of the repository. Where to get each repository, the commands, and the exact output of every run are gathered at the end of the article.

What each tool catches#

The first three rows are the common ground: every scanner finds private keys, cloud and API keys, and session tokens. The rows below are where the tools differ. Each one is a type of secret that turned up in at least one codebase and that only some tools reported. The pattern is steady. Once a secret stops matching a recognizable format, the text-scanning tools tend to miss it.

Results, one codebase at a time#

In each chart, the length of the bar is the number of findings. The color shows what those findings turned out to be once checked. A long bar is not automatically good: on a clean codebase it just means more false alarms to sort through. Where detect-secrets produced far more findings than the others, its bar is cut off and faded, with the real number shown beside it.

False alarms and speed#

Finding secrets is only half of the job. The other half is how much unrelated material you read through to act on the real findings. The clean WordPress release is the clearest measure of this. With no secrets to find, every result is a false alarm, and the range ran from zero for Trestle to 180 for detect-secrets. The noise comes from one practice: flagging anything that looks random. That accounts for almost all of detect-secrets' 7,700 findings on the demo project and 2,114 on Juice Shop, because the long encoded strings in notebooks and compiled files look random whether or not they hold a secret.

On speed, Trestle was the fastest on three of the four codebases and nearly tied on the fourth, the 66 KB planted-credentials set, where Gitleaks was faster by about 0.02 seconds. The gap grows with size. On the WordPress release, Trestle finished in about 0.7 seconds against roughly 13 seconds for Gitleaks. detect-secrets was the slowest throughout, taking up to about 51 seconds on Juice Shop.

Which tool to use#

If the secrets you care about are keys, cloud credentials, and tokens, any of these four tools will find them. Choose based on speed, ease of use, and how the output fits your workflow. On the clean release, Gitleaks and TruffleHog stayed nearly silent, with two false alarms each. detect-secrets is built to be used with a saved baseline that you review and approve once; run without one, its randomness checks produce a lot of noise.

Most codebases hold more than keys. They hold hashed passwords, database passwords that are ordinary words, payment data in test files, recovery phrases, and settings that get sent to the browser. These are the types a text-pattern scanner cannot recognize and a randomness check cannot catch. Across every codebase in this test, Trestle was the only tool that reported them, while staying silent on the clean release and finishing fastest on the larger sets. That is the practical effect of reading values out of the code instead of scanning text.

The results are worth checking rather than taking on trust. The repositories are public, the commands are four lines, and the points where the tools disagree are the most useful part. Download them and run the test.

How to reproduce this#

This section has everything needed to repeat the comparison: where to get each repository, the commands, and a sample of the output of every run. Run the four commands at the top of each repository and compare against the output shown.

# the same four commands, run at the top of each repository
trestle --explain
gitleaks dir -v
trufflehog filesystem .
detect-secrets scan

--explain is a Pro flag, and it is not required to reproduce these results. It adds a plain-language explanation and fix steps to each finding. Plain trestle reports the same findings without it.

The four repositories#

What each command printed#

The full output of each tool. Select a tool, then a repository.

Loading output...