Guide
Does Hiding Text in a PDF Remove It? A Test on 2009-Era US Government Documents
By Happy Turtle Tech ยท Published
We took one line of text from page 1 of each of 200 PDFs in a fixed sample of 2009-era US government documents, hid it in seven ways, and tried to get it back. Six of the ways we built ourselves from what they do to the file; the seventh is HerePDF's own Redact tool, run for real in a browser. A black rectangle, a white rectangle, a markup annotation, that annotation flattened into the page, and a rewrite of the whole file by a different PDF writer each left the text recoverable in 200 of 200 files. Turning every page into a picture left it in 0 of 200. Redact left it in 0 of 199.
This describes this sample and these methods. It is not a claim about PDFs made today, about how often people redact badly, or about any program we did not run. Along the way the study found a real gap in our own tool, which is fixed; the section on it says what it was.
What we did
The files come from GovDocs1, a public research collection of web documents published by US federal agencies in 2009. It is the same sample as our metadata study, which lists the checksum of every zip. Of its 4,840 PDFs, 3,288 could be used: the file had to open without a password, have no encryption of any kind, run to 30 pages or fewer, be under 5 MB, and have a line of text on page 1 that is at least 8 letters and digits long and appears nowhere else on that page. The other 1,552 were left out (more than 30 pages: 955; no suitable line: 246; encrypted: 192; no text on page 1, such as a scan: 114; too large: 45). From the 3,288 we took the 200 whose names have the lowest SHA-256, a rule with nothing to tune, fixed before any result was seen.
In each file the first line that met the conditions was the text to hide, and the box was its position on the page. The ways of hiding it:
- Shapes and annotations: a black or white rectangle drawn into the page, a black markup annotation left as an annotation, and the same annotation flattened into the page.
- A rewrite: the black rectangle, then the whole file written out again by Ghostscript, a different PDF writer. It stands in for a Save As or print-to-PDF step; we did not test those programs.
- Pictures: every page rendered to a 150 dpi picture by Ghostscript and put into a new PDF.
- Redact: the built site, in Chromium. The file is opened in the tool, a box is dragged with the mouse over the text on page 1, and the result is downloaded. The default 150 dpi was used.
How we tried to get the text back
A file counts as recovered if any of these found the text: reading the page the way copy and paste or search does (PDF.js); finding one of the original page 1 content streams still in the file, byte for byte, unless another page of the original also drew it; or finding a readable copy in something only page 1 can reach, or in an object nothing points to. The same words on another page, in the document properties or in a bookmark are counted separately in the second table, because the question there is different.
A clean result from a broken output proves nothing, so every output was also checked: it opens, it has the original number of pages, and the box covers the text. That last check renders page 1 with Ghostscript, independent of the reader that looks for the text, and measures how much of the box is dark (light for the white rectangle). A box that missed would show up there.
What was recovered
| What was done to the text | Text recovered | Could be copied from the page | Original page content still in the file | Box covered the text |
|---|---|---|---|---|
| Nothing done (the control) | 200 of 200 | 200 of 200 | 200 of 200 | n/a |
| A black rectangle drawn over the text | 200 of 200 | 200 of 200 | 200 of 200 | 200 of 200 |
| A white rectangle drawn over the text | 200 of 200 | 200 of 200 | 200 of 200 | 200 of 200 |
| A black rectangle added as a markup annotation, left as an annotation | 200 of 200 | 200 of 200 | 200 of 200 | 200 of 200 |
| That annotation flattened into the page | 200 of 200 | 200 of 200 | 200 of 200 | 200 of 200 |
| The black rectangle, then the whole file rewritten by a different PDF writer (Ghostscript) | 200 of 200 | 200 of 200 | 0 of 200 | 200 of 200 |
| Every page turned into a picture by Ghostscript | 0 of 200 | 0 of 200 | 0 of 200 | 200 of 200 |
| HerePDF Redact: the real tool, in a real browser | 0 of 199 | 0 of 199 | 0 of 199 | 199 of 199 |
Redact was run on 199 of the 200 files. 1 was refused (see the FAQ): one page about the size of a poster is too large to draw at 150 dpi in a browser, and the tool says so instead of producing a blank page. The other columns for the Redact row are over the 199 it processed.
The untouched files are the control: the text was recoverable in 200 of 200, so the checks can find it when it is there. Rewriting the file with another PDF writer changed how the page is stored, so the original streams were gone in 200 of 200, but the text was still on the page and could still be copied.
What turning pages into pictures costs
The two methods that removed the text did it by making the page a picture. That has a price: the rest of the text on that page can no longer be selected or searched, and the file grows.
| Method | Page 1 still has selectable text | File size against the original (middle, 9 in 10 files at or below, largest) |
|---|---|---|
| Every page turned into a picture by Ghostscript | 0 of 200 | 5.65 times, 17.65 times, 29.18 times |
| HerePDF Redact (only the pages you mark) | 0 of 199 | 2.65 times, 8.5 times, 36.92 times |
| A black rectangle, then a Ghostscript rewrite | 200 of 200 | 1.07 times, 2.07 times, 3.34 times |
Redact only does this to the pages you mark; the others keep their text. The rewrite row is there for comparison: it is the same file with its text intact.
The same words in other places
Hiding a line on one page does not touch other copies of the same words. In the untouched files the words also appeared elsewhere in the cases below, and this is what Redact did to them:
| Where else the words were | Untouched files | Still there after Redact | What the tool says it does |
|---|---|---|---|
| The same words on another page | 72 of 200 | 72 of 72 | Not changed (the tool says body text on pages you do not mark is left alone) |
| In a bookmark title | 5 of 200 | 0 of 5 | Replaced |
| In a comment | 0 of 200 | 0 of 0 | Removed or replaced |
| In the document properties or XMP | 17 of 200 | 17 of 17 | Not changed (the Metadata tool is the one for these) |
After Redact, 49 of 199 (24.6%) of the processed files still held a readable copy of the words somewhere. In every one of them it was on another page or in the document properties, which the tool states it leaves alone, and never on page 1, in a bookmark or in an orphaned object. No file had the words in a comment, so the comment cleanup is covered by tests with generated files only. If the same words matter on those pages, mark those pages too, and clear the properties with the Metadata tool.
What we found in our own tool
Redact removes a tagged PDF's structure tree, because the alternate text and replacement text in it can quote the page. The first complete run of this study reported nothing left. Then we looked at the one file where a bookmark could still reach a copy of the words, and found that the bookmark was not holding a title: it was linked to a structure element, which kept that element, its alternate text, and the tree above it in the saved file, with the redacted words in it. Before the fix, 1 of 199 processed files did this. Our own first count had missed it, because it filed a copy reachable from a bookmark as something the tool leaves alone, when the tool promises to scrub bookmarks.
The tool now empties every structure element and removes every link to one, from bookmarks and from links alike, and a bookmark or link keeps its title and where it goes. Tests with generated tagged PDFs, naming an element through a bookmark, a link or a key the tool does not know, failed before the change and pass now, and we ran the same files again on the fixed site. The numbers above are those: 0 of 199 processed files had a copy left on page 1, in a bookmark, in a comment or in an orphaned object.
What this does not show
- It is documents published by US federal agencies in 2009. Other sources and today's software can differ.
- One line of text on page 1 of each file was hidden. Longer passages, other pages and other fonts were not tested.
- The cover-up methods are ones we built from what they do to the file. Nothing here says how Acrobat, Preview, Word or any other program behaves.
- Scanned pages have no text to find, so they were left out. A picture of text is not searched by any of these checks.
- Redact was run at its default 150 dpi. The sample had no rotated page, so rotation is covered only by tests with generated files.
- Short words are only matched by the tool as part of a longer phrase, and nothing here tests that limit.
- Password-protected and encrypted files were not read.
Check your own file
The guide on why hiding text usually does not remove it explains the idea; this page puts numbers to it. After redacting, search the result for the words you meant to remove and try to select the text under the box. The checklist before sending a PDF covers what else a file can carry, and how we test describes the whole-file scan these results use. Cropping has the same limit as a box: what stays in the file after a crop.
Repeat it
The data file for this study holds every count above with its denominator and the versions of the tools used, with no file name, path or text from any document. The files are the GovDocs1 zips listed in the metadata study's data file, and the sample is the 200 that the stated rule picks. The file also holds a checksum of the sample's list of names, so a rerun can confirm it chose the same files. We do not publish our scripts, so the Redact column can only be repeated by running the tool yourself.
Everything on HerePDF runs in your browser. Your files are not uploaded, before or after you use these tools.
Frequently asked questions
Is drawing a black box over text in a PDF enough?
In this test it was not. We drew a black rectangle over one line of text in each file and the text could still be copied out of the page, and the original page content was still in the file. The same was true of a white rectangle and of a markup annotation. The page looks covered because the rectangle is painted on top; the text underneath is a separate instruction that is still there.
Does HerePDF Redact remove the text from the file?
In this test it did. We drove the real tool in a real browser, drew a box over the line with the mouse, saved the result and looked for the text by reading the page, searching the file for the original page content, and checking every place that could hold a copy. What the tool states it leaves alone, such as the same words on other pages you did not mark and the document properties, are reported separately on this page.
Did you test Adobe Acrobat, Preview or Word?
No. We built each way of hiding text from what it does to the file: a shape on the page, an annotation, a rewrite by another PDF writer, a page made into a picture. Nothing on this page says how any named program behaves.
Does this show how often people redact badly?
No. We applied each method ourselves to files from a fixed sample. The study measures what a method leaves in the file, not how often people use it or get it wrong.
Why did Redact refuse one of the files?
One page was about the size of a poster, and drawing it at the tool's default resolution would pass the limit of a browser canvas. The tool says so and does nothing, rather than produce a blank page with black boxes. That file is reported as refused, not as passed.