To de-identify clinical notes before an LLM sees them, layer your detectors and score recall for each identifier separately. I learned that the hard way. The first regex-only version of MedScrub's de-identifier scored 71% recall overall on our benchmark. That number hid two zeros. Medical record numbers and Social Security numbers were both at 0%, and those are the two you'd least want a model to see.
Here's the short version. Never blend recall into one number. Make the build fail when a structured identifier drops below 100%. Decide what happens to dates on purpose, because Safe Harbor and a usable clinical timeline want opposite things.
Everything below comes from building that layer into MedScrub, plus smaller versions in MedHook and eSpiral.
What a regex PHI scrubber misses in clinical notes
That first benchmark was tiny: three synthetic notes and 14 labeled identifiers. Recall came out at 0.714. Besides the MRN and the SSN, it missed a facility name and a first-name-only mention in the narrative. Names overall were at 75%.
We knew exactly what was missed because the benchmark prints every miss with the text around it. That one feature did more for us than the summary score.
The SSN miss came from our own code. Our SSN validator rejected the fake test SSN as invalid, so the detector never fired. The README called that "correctly rejected." I don't buy it. A validity check is a precision tool. A real note can contain a mistyped SSN, and anyone reading it still sees an SSN. HHS says a listed identifier has to be removed wherever it appears in a record if it is recognizable as an identifier, free text included.
The MRN miss had two explanations, and they disagreed. The README blamed "standalone numbers without label." But the MRN missed in the results file had a label. Had we trusted the summary, we'd have fixed the wrong thing.
The fix list was boring and correct. Add MRN patterns. Debug why SSN detection existed but wasn't firing. Add label-based detection for cues like SSN:, DOB: and Patient:. The bigger lesson was that patterns alone would never get there. We moved to layers: label-aware patterns, then NER, then a model we trained ourselves. It's the same reason I argue for building a GenAI platform in layers: each layer catches what the one before it can't, and you can measure each one.
Three notes is not a benchmark you'd publish. It was still enough to find two zeros. Start small, the way you'd start any product eval, and grow the set once it stops surprising you.
Measure recall per PHI type, not overall
A minimal harness takes gold spans and predicted spans for a note, then returns recall per type and every miss with its context. It's the same idea as any honest LLM eval: a number you can trust, plus the failures you can read.
from collections import defaultdict
def recall_by_type(gold, predicted, text, ctx=30):
"""gold, predicted: lists of (start, end, phi_type) spans for one note."""
hits, totals, misses = defaultdict(int), defaultdict(int), defaultdict(list)
for start, end, phi_type in gold:
totals[phi_type] += 1
if any(ps <= start and pe >= end for ps, pe, _ in predicted):
hits[phi_type] += 1
else:
I'd bake in two choices. First, count a gold span as caught if any predicted span covers it, whatever label the detector gave it. For redaction, an MRN tagged as a phone number is still gone. Second, return misses as text, not counts.
Recall matters more than precision here. One missed identifier is a breach. An over-redacted word gives you a worse answer. Our own MedScrub notes say native NER over-redacts clinical terms, so the cost is real. How much over-redaction you count also depends on your gold labels. HHS says free text describing a patient as "Executive Vice President of the state university" must go. A stand-alone "Newark, NJ" in intake notes, with no clear link to the patient's address, may stay. A recall-first tool strips Newark either way. Decide in your labeling guide whether that counts as an error.
Today MedScrub's Safe Harbor benchmark scores each identifier separately. Twelve structured categories and five narrative ones must hit recall of exactly 1, MRN and SSN among them. The test carries a comment: "do NOT weaken this assertion." It exists so that when the test goes red, the fix goes into the detector, not the threshold.
Separately, our current model's training log reports recall of 0.9990, precision of 0.9992 and F1 of 0.9991. That's a training metric, not the gate. The gate is the Safe Harbor benchmark, and its corpus is synthetic. Our own docs say plainly that it is not a legal determination. Passing it is a regression check, nothing more.
How to benchmark regex vs Presidio vs a local LLM on your own notes
I don't have Presidio or LLM-pass recall on my own notes, and I won't make numbers up. This is the method, so you get real per-type numbers on your notes instead of trusting a vendor benchmark of the vendor's own product.
The first mistake with Presidio is assuming it knows what an MRN is. The current supported-entities list has healthcare US recognizers for NPI, MBI, health insurance member ID, claim number, prescription number and a few more. It has no generic medical record number entity. So you write that one yourself, then adapt Presidio's output to the harness:
from presidio_analyzer import AnalyzerEngine, Pattern, PatternRecognizer
mrn = PatternRecognizer(
supported_entity="MRN",
patterns=[Pattern("mrn_labeled",
r"\b(?:MRN|Medical Record (?:No\.?|Number|#))\s*[:#]?\s*[A-Z0-9-]{5,12}\b", 0.6)],
context=["mrn", "medical record", "chart"],
)
analyzer = AnalyzerEngine()
analyzer.registry.add_recognizer(mrn)
def presidio_spans(note, threshold=0.3):
return [(r.start, r.end, rentity_type
That low threshold is deliberate. In a recall-first pipeline, every score cutoff is a recall decision, so set it while looking at the harness output.
Then score each configuration on the same labeled notes, one at a time:
- Your regex baseline.
- Stock Presidio plus the MRN recognizer.
- That, plus clinical NER. Presidio's demo loads
obi/deid_roberta_i2b2 and StanfordAIMI/stanford-deidentifier-base through TransformersRecognizer.
- That, plus an LLM pass. presidio-analyzer can call a local model through Google's LangExtract with Ollama (
pip install "presidio-analyzer[langextract]", BasicLangExtractRecognizer). Its results come back through the same analyze call, so presidio_spans feeds them to the harness unchanged. Presidio's own docs label this path an "Experimental Feature". Treat it that way.
Pin the Presidio version, and the model name and version for steps 3 and 4, and record them next to the results. An older copy of the microsoft/presidio docs lists only general US IDs, without the healthcare recognizers. Check what your installed version actually ships.
Where should the LLM pass earn its place? MedHook's agent scrubber has a documented blind spot. When a name is typed with no cue word, as in "send Maria's labs," patterns can't tell it apart from a field name. So the MedHook UI says common identifiers are masked, not everything. When you can't measure the rest, that's the honest claim. Bare names in narrative are where a model that reads context should help, along with HHS's own ambiguity example: is "5/97" a date or a ratio? A local model keeps raw text inside your boundary, which is one reason small models are often the right call in healthcare. It still sits behind the same assertions. If it can't hold MRN at 1.0 alongside the patterns, it's an extra layer, not a replacement.
Date shifting vs Safe Harbor: pick one on purpose
Safe Harbor is blunt about dates. Every date element except the year goes. "January 1, 2009" becomes "2009." Lab report dates count too. Ages over 89 are grouped as 90 or older.
Clinical care runs on dates. I've never shifted dates in any of these products. We made a different, explicit tradeoff in each one.
In MedScrub, the generic path used to tokenize every date. Chart answers became useless, because physicians need the timeline. So we pinned a DOB-only policy for structured data. Encounter, onset and recorded dates stay real. Birth dates and dates written in narrative get hidden.
HHS's own numbers show why the birth date is the one to hide. Year of birth, sex and three-digit ZIP is unique for about 0.04% of US residents. Full date of birth, sex and five-digit ZIP is unique for over 50%. A birth date follows a person everywhere. An encounter date is one event.
That still isn't Safe Harbor, and we don't pretend it is. Our benchmark test calls it a "known, deliberate policy gap" against the Safe Harbor date rule, and asserts that date recall is below 1 but above 0. If strict Safe Harbor is ever required, that assertion flips.
In eSpiral we went the other way. Problem dates sent to the model are cut to the year. That was cheap, because the model only needs rough history there.
What about shifting every date by a per-patient offset? It keeps intervals intact. But the HHS guidance I've read never endorses it, and Safe Harbor allows no date element except the year. My read is that a date-shifted note is an Expert Determination question, not a Safe Harbor one. HHS doesn't say that outright. I still wouldn't bet a compliance posture on the opposite.
Surrogate tokens and re-identification maps: can you hash patient identifiers?
Safe Harbor sets rules for re-identification codes. A code must not be derived from the patient's information, must not be translatable back to them, and the mechanism can't be disclosed. HHS also says a hash with no secret key counts as an identifying code. So don't hash the MRN to make a token.
Use random tokens, keep them consistent within a document so the note still reads correctly, and keep the map on your side of the boundary. A minimal version:
import secrets
class SurrogateMap:
"""One per document. Stored in your service, never in the prompt."""
def __init__(self):
self.forward, self.reverse = {}, {}
def token(self, value, phi_type):
key = (phi_type, value)
if key not in self.forward:
tok = f"[{phi_type}_{secrets.token_hex(4)}]"
self.forward[key selfreversetok tok value
The model sees [NAME_3f9a1c2e] every time the same patient comes up, so pronouns and references still line up. The real values only come back in reidentify, on your side.
Under Expert Determination, keyed cryptographic hashes of PHI can be disclosed if an expert approves and the keys stay private. Different method, different design, so choose the method first.
Fail closed: where the boundary actually breaks
The boundary breaks in boring places.
In MedScrub, raw names once reached our hosted Neon database, which broke the no-BAA posture we'd designed around. We fixed it. That incident is why I distrust any "the model never sees PHI" claim that doesn't have a test behind it.
Dates can leak on the way back, too. MedScrub date tokens once leaked raw at re-identification time for a design partner. That path has a test now. Re-identification is code like any other, and it needs the same coverage as the scrub step.
eSpiral had a subtler failure. The scrub step could alter the batch separator, which meant history text could fall back unscrubbed. We made it throw instead.
MedHook leans the same way. HL7v2-to-FHIR conversion drops the SSN in PID-19 unless you opt in with includeSsn. A timestamp with no offset and no configured timezone fails rather than being read as UTC.
A scrubber that errors costs you a retry. A scrubber that passes text through costs you a breach.
Safe Harbor or Expert Determination for LLM pipelines?
HHS recognizes exactly two de-identification methods. For a pipeline, a few details change your code.
Safe Harbor punishes partial identifiers. Initials fail. The last four digits of an SSN fail. A three-digit ZIP only survives if every ZIP sharing that prefix adds up to more than 20,000 people, otherwise the prefix becomes 000. HHS lists 17 restricted prefixes based on 2000 Census data (036, 059, 063, 102, 203, 556, 692, 790, 821, 823, 830, 831, 878, 879, 884, 890, 893) and warns not to rely on that list if newer data exists. That's a lookup table, not a regex: zip3 = "000" if zip5[:3] in RESTRICTED_ZIP3 else zip5[:3].
Expert Determination has no fixed numeric threshold. An expert defines a "very small" risk for the expected recipient and documents the methods and results.
My rule: if your model needs real clinical dates, you are not doing Safe Harbor. Say so in a test, the way we did. Then either bring in an expert or tell customers exactly what you keep. If the model only needs rough history, generalize to the year and take Safe Harbor's simplicity. Consent and access are separate questions, covered in HIPAA Consent: What Engineers Actually Need to Know.
If you do one thing this week, split your recall number by identifier type and make MRN and SSN assert at 1.0. That's how we found our zeros.
