knowledge base

Everything about DeIdentify, in plain language

Written for anyone — whether you've never touched SQL or you run nightly ETL. Use the sidebar to jump around, or search for a specific term.

Getting Started

What is DeIdentify?

DeIdentify is a tool that takes a database export (SQL or CSV) and replaces the personal information inside it with realistic-looking fake data. It runs entirely in your browser — nothing is uploaded anywhere.

You get an output file with the same structure (same tables, same columns, same row counts, same relationships between tables) but with the sensitive values replaced. That file is safe to share with developers, analysts, or vendors who shouldn't see the real data.

infoNever worked with SQL before? A .sql file is just a text file that describes a database. Open it in Notepad and you'll see human-readable commands. DeIdentify handles all the parsing for you.

When should I use it?

  • You need to give a production data snapshot to a developer or analyst who isn't cleared for real PII.
  • You're setting up a staging or QA environment and want realistic data without the compliance risk.
  • You're sending a bug report to a vendor and want to strip identifiers from a data sample.
  • You're preparing a dataset for internal training or a demo.
  • You need de-identified data for research under HIPAA Safe Harbor or Expert Determination.

5-step quick start

  • Drag a .sql or .csv file anywhere onto the page (or click 'Choose file').
  • Review the auto-detected columns. Sensitive fields are highlighted; the tool guesses what each one contains (email, phone, name, etc.).
  • Optional: click the Preset menu and pick 'HIPAA Safe Harbor' if this is health data — this configures every column at once.
  • Click the green 'De-identify' button. You'll see a summary of what changed and a Diff view.
  • Click 'Download' (or 'Copy') to save the rewritten file. Optionally download the Audit report for your records.
tipNo file to try? Click 'Load sample' on the home page — it loads a small fake healthcare dataset so you can see the whole flow.

Core Concepts

SQL & CSV in one minute

A SQL dump is a text file with two kinds of statements you care about:

CREATE TABLE patients (
  id INT PRIMARY KEY,
  first_name VARCHAR(100),
  email VARCHAR(255)
);

INSERT INTO patients (id, first_name, email) VALUES
  (1, 'Alice', 'alice@example.com'),
  (2, 'Bob',   'bob@example.com');

CREATE TABLE defines a table (name + columns). INSERT INTO ... VALUES adds rows. DeIdentify reads the CREATE TABLE to learn the shape, then rewrites the values inside INSERT lines.

A CSV file is even simpler — the first row is column names, the rest are values separated by commas. Same idea, one table per file.

How columns are detected

DeIdentify looks at each column two ways:

  • By name — 'email' → email, 'first_name' → firstName, 'ssn' → ssn, 'mrn' → medical record number, and dozens more.
  • By value — if the first 30 non-null values match a known pattern (like nnn-nn-nnnn for SSN, or an email format), it's flagged even if the column has a weird name.
  • By your custom rules — you can add regex patterns that override or extend the built-in list.
infoThe detected type is a guess. Review the column plan before running — you can change any column's strategy in one click.

Referential integrity — the important bit

If patients.id = 42 belongs to Alice, and encounters.patient_id = 42 is Alice's visit, then after de-identification both need to become the same new value (say, 7891). Otherwise the visit is orphaned.

DeIdentify handles this automatically when: (a) the primary key column is in your CREATE TABLE, (b) INSERT statements include the primary key value explicitly, and (c) foreign-key columns follow the '<table_singular>_id' convention or you set the link manually.

noteIf you export your dump with auto-increment IDs replaced by DEFAULT (common in pg_dump without --inserts), primary keys aren't in the file — the preflight check will flag this so you can re-export.

Strategies (per column)

Keep

Leave the value exactly as-is. The output column is byte-for-byte identical to the input.

When to use: columns you have personally confirmed contain no direct identifiers, quasi-identifiers, or free-text that could leak PII — e.g. product SKUs, status flags, boolean columns, category enums, non-personal system timestamps like created_at on a lookup table.

Risk: Keep is the only strategy that can leak data if you're wrong about the column. When in doubt, choose Fake or Redact instead.

Fake (realistic)

Replace each real value with a synthetic value of the same shape and type. A name becomes a plausible name, an email becomes a syntactically valid email, a phone number stays formatted like a phone number.

Deterministic within a run: the same input plus the same salt always produces the same fake value, so joins across tables continue to work (customer 'Alice Smith' becomes 'Marta Hoffman' in every table she appears in). Change the salt to get a completely new mapping.

When to use: this is the default and best choice for most identifying columns (names, emails, phones, addresses, MRNs). It preserves data shape so applications, forms, and reports keep working against the de-identified copy.

Not reversible without the original mapping table (which DeIdentify never stores).

Hash (pseudonym)

Replace the value with a short deterministic token derived from HMAC(salt, value) — for example, a1f9c3d2. Same input plus same salt always produces the same token.

When to use: internal identifiers where you need uniqueness and referential integrity but don't need the output to look human — foreign keys, session IDs, external system IDs, join keys between tables.

Security: not reversible without brute-forcing the salt against a known value space. Keep the salt private if you don't want a knowledgeable insider to re-link tokens to originals.

Shift date ±

Add or subtract a random offset (in days) from every date/time value. Intervals between events are preserved — if admission was 4 days before discharge in the source, it is still 4 days apart in the output.

  • Per-entity (recommended for clinical data): every date belonging to one patient/subject is shifted by the same offset. Different patients get different offsets. Preserves each individual's timeline while making the absolute calendar meaningless. Required style for HIPAA Expert Determination workflows.
  • Per-table: every date in the table is shifted by the same offset. Simpler and faster, but weaker protection when a table is dominated by one subject.

When to use: encounter dates, appointment times, transaction timestamps, event logs — any temporal data where you want to preserve durations and sequence but hide the actual calendar.

Generalize

Replace a precise value with a broader bucket. Reduces uniqueness (and therefore re-identification risk) while keeping the column useful for analytics.

  • Age 78 → '75-84' age band
  • Date 2024-03-18 → 2024 (year only) or 2024-03 (month)
  • ZIP 10024 → 100XX (ZIP3, the HIPAA Safe Harbor rule for geography)
  • Salary 87,412 → '80k–90k' band

When to use: quasi-identifiers where the exact value isn't needed downstream. This is the HIPAA Safe Harbor approach for dates and geographic detail and the primary tool for reducing k-anonymity risk.

Redact / Null

Remove the value entirely. The output cell is written as NULL (SQL) or empty (CSV). No trace of the original remains in the file.

When to use: columns with no analytic value to the recipient, or free-text fields where you can't guarantee what's inside — clinical notes, internal comments, password hashes, security answers, uploaded document blobs, columns that consistently contain names or SSNs mixed with other text.

Strongest guarantee of the six strategies — you cannot leak what isn't there. Use liberally when in doubt.

Study ID (SID-XXXXXXXX)

Replace each real value with a deterministic study identifier of the form SID-XXXXXXXX (a configurable prefix plus a 10-character token derived from FNV-1a of salt + column + original value). Purpose-built for research and backend-migration workflows where you need a stable join key you can reproduce later.

Deterministic across sessions: the same salt + same column + same original value always produces the same Study ID, on any machine, in any browser, at any point in the future. Save your salt (and any per-column prefix you customized) and you can re-run the same de-identification months later and get byte-identical IDs — that is what makes cross-database mapping possible.

Linkable across tables: like Hash and Fake, Study ID participates in referential-integrity linking. If encounters.patient_id is linked to patients.id and both use Study ID with the same prefix, the same subject gets the same SID in every table, so joins keep working end-to-end.

Per-column prefix: change the prefix in the column plan to distinguish entity types at a glance — e.g. PT- for patients, ENC- for encounters, PROV- for providers, SPEC- for specimens. Prefix is capped at 12 characters and is not itself hashed.

When to use: (1) cohort/study identifiers where you want a clean opaque token instead of a random-looking hash; (2) test → prod migration where you must map a de-identified record back to the real subject after validation; (3) any multi-database or multi-team workflow that needs a shared stable ID without exchanging PHI.

Reversibility: Study ID is one-way on its own — you cannot derive the original value from the SID. Reversibility requires that you keep the original source data (or a separately-stored, secured mapping) and re-run Study ID with the same salt + prefix to reproduce the link.

Reproducibility checklist: to reproduce a Study ID later you need — the exact salt, the exact column name (case-sensitive), the exact prefix, and the exact original value. Export your profile (Custom Rules → Export profile) to capture all of these in one JSON file, or save a named preset in the browser (see Custom Rules → Saved presets & profiles).

Presets

HIPAA Safe Harbor

Applies the Safe Harbor rule: remove or generalize all 18 identifier categories that the HIPAA Privacy Rule requires. Result is de-identified under 45 CFR 164.514(b)(2).

  • Names, geographic subdivisions smaller than a state (except first 3 ZIP digits, and only if the geographic unit contains > 20,000 people)
  • All dates directly related to an individual (except year), and all ages > 89
  • Phone, fax, email, SSN, MRN, health plan number, account number
  • Certificate/license numbers, vehicle IDs, device serial numbers, URLs, IPs
  • Biometric identifiers, full-face photos, any other unique code
noteSafe Harbor is a rule, not a magic bullet. You are still responsible for reviewing that no free-text field or combination of columns re-identifies someone.

Expert Determination (workflow)

The tool doesn't certify — a qualified statistician does — but it gives you the levers: per-entity date shifts keep clinical intervals intact, fake generators preserve distributions, and hash keeps join keys working. Export the Audit report to hand to your reviewer.

42 CFR Part 2 (behavioral health)

Extra strict rules apply to substance use disorder records. DeIdentify recognizes provider_name, facility, diagnosis_code, and substance_use fields as sensitive by default and redacts or generalizes them.

Regulations & Requirements

HIPAA — what the rule actually requires

The HIPAA Privacy Rule (45 CFR §164.514) says PHI is 'de-identified' — and therefore no longer PHI — only if one of two methods is applied. DeIdentify supports both.

  • Safe Harbor (§164.514(b)(2)): mechanically remove or generalize 18 specific identifier categories AND have no actual knowledge that the residual info could re-identify anyone. Deterministic and testable.
  • Expert Determination (§164.514(b)(1)): a person with appropriate statistical/scientific expertise certifies that the risk of re-identification is 'very small' and documents the analysis. DeIdentify provides the mechanics (per-entity date shifts, generalization, hashed pseudonyms, audit reports) — a qualified expert makes the call.
infoPHI = individually identifiable health information held or transmitted by a covered entity or business associate. Once de-identified under either method, the data is no longer PHI and falls outside HIPAA's use/disclosure restrictions.

HIPAA Safe Harbor — the 18 identifiers

Every identifier in the following table must be removed for the record — and for the individual's relatives, employers, and household members — before the record qualifies as Safe Harbor de-identified. The right column shows how DeIdentify handles each when the Safe Harbor preset is applied.

#Identifier (per §164.514(b)(2))How DeIdentify handles it
1NamesDetected as firstName / lastName / fullName → Fake (realistic synthetic name, deterministic per salt)
2Geographic subdivisions smaller than state (street, city, county, precinct, ZIP)address / city → Fake · zipcode → Generalize to first 3 digits (100XX). Preset does not auto-suppress the 3-digit ZIP for the ~17 restricted low-population prefixes — review manually if you have those.
3All date elements (except year) directly related to an individual — birth, admission, discharge, death, and all ages > 89dob / date → Generalize to year only. age → Generalize; ages 90+ bucketed to '90+'.
4Telephone numbersphone → Fake
5Fax numbersfax → Fake
6Email addressesemail → Fake (syntactically valid, non-routable domain)
7Social Security numbersssn → Fake (format-preserving, not a real issued SSN)
8Medical record numbersmrn → Fake / Hash
9Health plan beneficiary numbershealthPlanId → Fake
10Account numbersaccountNumber → Fake
11Certificate / license numbers (incl. NPI, DEA, driver license)licenseNumber → Fake
12Vehicle identifiers and serial numbers, including license platesvehicleId → Fake
13Device identifiers and serial numbersdeviceId → Fake
14Web URLsurl → Redact
15IP addressesip → Redact
16Biometric identifiers (fingerprints, voice prints)Not auto-detected — must be Redacted manually or via a custom rule (typically stored as blobs)
17Full-face photographs and comparable imagesNot auto-detected — Redact any image / blob columns manually
18Any other unique identifying number, characteristic, or codeAdd a custom rule (scope: name or value) → target 'custom' → Hash or Redact
noteSafe Harbor also requires you have no actual knowledge that the remaining information could identify the individual. DeIdentify cannot make that judgment for you — review free-text notes, rare diagnosis combinations, and small-cell counts before releasing.

HIPAA Expert Determination — supporting workflow

Expert Determination is risk-based rather than rule-based. The expert typically wants to preserve analytic utility (dates, ages, ZIPs, diagnoses) while proving re-identification risk is very small. DeIdentify supports this with:

  • Per-entity date shifting — every date for one subject is shifted by the same random offset, so intervals (admit → discharge, dose → dose) stay exact but the calendar is meaningless. Different subjects get different offsets.
  • Deterministic pseudonymization — Hash strategy produces the same token for the same input+salt, so joins across tables remain valid without keeping real IDs.
  • Generalization — bucket ages, dates (year/month), ZIP (3-digit), salaries, etc. Reduces uniqueness for k-anonymity analysis.
  • Fake generators — preserve shape and distributional characteristics (name lengths, email domains, phone formats) so statistical utility is minimally affected.
  • Audit report (JSON + CSV) — per-column counts of transformed / redacted / kept values, plus hashed samples for spot-checking. Hand this to your expert.

42 CFR Part 2 — SUD treatment records

42 CFR Part 2 protects records of federally-assisted substance use disorder (SUD) treatment programs. It is stricter than HIPAA because the mere fact that a person received SUD treatment can itself cause discrimination or legal harm.

  • Applies to Part 2 programs: any individual or entity that holds itself out as providing, and provides, SUD diagnosis, treatment, or referral for treatment, and receives federal assistance.
  • 'Patient identifying information' under Part 2 includes not only direct identifiers but any info that would identify a patient as having/having had a SUD — including the name of the treatment facility itself.
  • Disclosure without patient consent is prohibited except for narrow purposes (medical emergency, audit, research with an IRB, court order). Even de-identified disclosures require care.
  • 2024 HHS final rule aligned Part 2 more closely with HIPAA for TPO uses but retained the identity-of-treatment protection.

How DeIdentify handles the Part 2-specific risk surface:

Field / signalDetectionHandling under Part 2 preset
provider_name, physician, clinician, therapist, counselor, attending, referring_providerColumn nameRedact — a SUD-program provider name identifies the program
facility, program, program_name, clinic, hospital, treatment_center, site, locationColumn nameRedact — the facility name itself is protected identifying information
diagnosis, dx, dx_code, icd10_code, icd9, dsm, problem_codeColumn nameRedact — SUD ICD-10 codes (F10–F19) are self-identifying
Values matching F10–F19 (any column)Value regexDetected as diagnosisCode → Redact
substance_use, sud_flag, drug_use, alcohol_use, addiction_flag, part2_flagColumn nameRedact — explicit SUD indicator
notePart 2 layers on top of HIPAA — use the 'HIPAA + 42 CFR Part 2 (strict)' preset for behavioral-health datasets. Even after de-identification, keep in mind that pattern-of-visits data (e.g. weekly appointments at a known SUD clinic) can re-identify.

PII — general (non-health) requirements

'PII' is defined by many statutes, not one. DeIdentify's PII (basic) preset targets the common categories used across NIST SP 800-122, US state privacy laws (CCPA/CPRA, VCDPA, etc.), GLBA for financial data, FERPA for education records, and GDPR's 'personal data' definition.

  • Direct identifiers: name, email, phone, SSN or national ID, government ID numbers, credit card / financial account, driver's license, passport, biometrics.
  • Quasi-identifiers (linkable): date of birth, ZIP + gender + DOB (the classic Sweeney combo), device IDs, IP addresses, cookies, precise geolocation, usernames.
  • Sensitive PII (heightened protection under many regimes): SSN, financial account, medical, biometric, precise location, children's data, race/ethnicity, religion, sexual orientation, immigration status.
  • Authentication material: passwords, password hashes, security question answers, MFA seeds, session tokens — always Redact.
infoThe 'PII (basic)' preset fakes direct identifiers, hashes usernames and UUIDs (preserves join keys), redacts passwords, and keeps dates unchanged. Layer HIPAA or Part 2 presets on top for health or SUD data.

Recognized Fields

Fields recognized by column name

On upload, every column name is matched against these regex patterns. First match wins. Custom rules run before the built-in list.

Detected typeColumn name patterns (case-insensitive)
Emailemail, e_mail, e-mail, email_address
First namefirst_name, fname, given_name, forename
Last namelast_name, lname, surname, family_name
Full namefull_name, display_name, customer_name, contact_name, patient_name, name
Phonephone, mobile, cell, tel, telephone, phone_number
Faxfax, fax_number
SSNssn, social_security, social_security_number
Addressaddress, street, street_address, address_line_1, addr
Citycity, town, locality
Statestate, province, region
Zipzip, zip_code, postal_code, postcode
Countrycountry, country_code
Date of birthdob, birth_date, birthday, date_of_birth
Dateadmit_date, admission_date, discharge_date, visit_date, encounter_date, service_date, death_date, date
Ageage
IP addressip, ip_address, remote_ip, client_ip
Credit cardcredit_card, cc_number, card_number, pan
Usernameuser, username, login, handle
Passwordpassword, pwd, passwd, password_hash
Companycompany, organization, org, employer, business_name
URLurl, website, homepage, link
UUIDuuid, guid
MRNmrn, medical_record_number, patient_number, chart_number
Account #account_number, acct_number, acct_no
Health plan #health_plan_id, health_plan_number, member_id, insurance_id, policy_number
Device IDdevice_id, device_serial, serial_number, device
License #license_number, dl_number, driver_license, npi, dea_number
Vehicle IDvin, license_plate, plate_number, vehicle_id
Provider name (Part 2)provider, provider_name, physician, clinician, therapist, counselor, attending, referring_provider
Facility / program (Part 2)facility, program, program_name, clinic, hospital, treatment_center, site, site_name, location, location_name
Diagnosis code (Part 2)diagnosis, diagnosis_code, dx, dx_code, icd10_code, icd9_code, dsm_code, problem_code
Substance-use indicator (Part 2)substance_use, sud_flag, drug_use, alcohol_use, addiction_flag, part2_flag

Fields recognized by value content

When the column name doesn't match, DeIdentify sniffs the first ~30 non-null values. If they all match a known shape, the column is tagged.

Detected typeValue pattern
Emaillocal@domain.tld
SSNNNN-NN-NNNN
Phone+? digits with spaces / dots / dashes / parens, ≥ 8 digits
Credit card4-4-4-4 (or 4-4-4-3) digit groups
IP addressN.N.N.N (IPv4)
URLhttp:// or https:// prefix
UUID8-4-4-4-12 hex
DateYYYY-MM-DD, YYYY-MM-DD HH:MM(:SS), or M/D/YYYY
Diagnosis code (SUD)F10–F19 (ICD-10 mental/behavioral disorders due to psychoactive substance use)
infoAnything not matched is tagged 'Unknown' and defaults to Keep — you decide what to do. Add a custom rule (scope: value) with a regex to catch project-specific patterns like internal IDs.

How each detected type is transformed

Detected typeDefault Fake outputNotes
Emailgiven.family####@example.orgDeterministic per salt; safe non-routable domain
First / Last / Full nameRealistic synthetic name from a curated poolSame input → same output within a run
Phone / Fax(NNN) NNN-NNNNFormat-preserving
SSNNNN-NN-NNNN, never a real issued SSNFormat-preserving; not reversible
Address / City / State / Zip / CountryRealistic synthetic address componentsZip may be Generalized under Safe Harbor
DOB / DateShifted or Generalized (year only)Per-entity shift preserves per-subject intervals
AgeBucketed (10-yr band; 90+ collapsed)Safe Harbor rule for ages > 89
IP / URLRedacted under Safe Harbor / Part 2Or Faked under PII (basic)
Credit cardLuhn-valid synthetic PANFormat-preserving
Username / UUIDHashed to short deterministic tokenPreserves joins
PasswordAlways RedactedNever Faked or Kept
Company / URLSynthetic company / non-routable URL
MRN / Account # / Health plan # / Device ID / License # / Vehicle IDFormat-preserving synthetic identifierOr Hash if used as a join key
Provider / Facility / Diagnosis / Substance-useRedacted under Part 2 presetsMarked as SENSITIVE_TYPES in the engine
UnknownKeep (unchanged)Add a rule or change the column strategy manually

Preset Behavior

What each preset does, side by side

Presets are one-click bundles that assign a Strategy to every detected column type. You can always override any column after applying a preset.

Detected typeHIPAA Safe Harbor42 CFR Part 2 (SUD)HIPAA + Part 2 (strict)PII (basic)Keep everything
Names, Email, Phone, Fax, Address, City, State, CountryFakeFakeFakeFakeKeep
SSN, MRN, Account #, Health plan #, Device ID, License #, Vehicle ID, Credit card, CompanyFakeFakeFakeFakeKeep
ZipGeneralize (3-digit)Generalize (3-digit)Generalize (3-digit)FakeKeep
Date, DOBGeneralize (year)Shift (per-entity by default)Generalize (year)KeepKeep
AgeGeneralize (bucket; 90+ collapsed)GeneralizeGeneralizeFakeKeep
IP, URLRedactFakeRedactFakeKeep
Username, UUIDFakeFakeFakeHash (preserves joins)Keep
PasswordRedactRedactRedactRedactKeep
Provider name, Facility, Diagnosis code, Substance-use (Part 2 sensitive)FakeRedactRedactFakeKeep
UnknownKeepKeepKeepKeepKeep
infoThe 'HIPAA + 42 CFR Part 2 (strict)' preset is the recommended default for any behavioral-health, SUD, or mental-health dataset. Use plain 'HIPAA Safe Harbor' for general clinical data with no Part 2 exposure, and 'PII (basic)' for non-health apps.
notePresets do not touch free-text columns (notes, comments, chart_text) because content varies. Review those columns manually — set Redact or add a custom value-scope rule.

Custom Rules

Adding a custom rule

Custom rules let you extend detection with regex patterns for column names or values. Click '+ show custom rules' in the toolbar.

  • Label: a short name for your rule (visible in the summary).
  • Scope: 'name' matches the column name, 'value' matches the actual values in the column.
  • Pattern: a regular expression. Wildcards are just regex (.* matches anything).
  • Target: what type this column should be treated as (email, mrn, custom identifier, etc.). Determines the default strategy and fake generator.
Examples:
  ^emp_id$              scope: name    target: accountNumber
  patient.*number       scope: name    target: mrn
  ^EMP\d{6}$            scope: value   target: username

Saved presets & profiles

DeIdentify gives you two ways to save a data configuration so you can come back to it later — one lives in your browser, one lives in a file you control.

Saved presets (in-browser): on the data page, open the Preset dropdown and click '+ Save current column plan as preset…'. Give it a name and it is stored in this browser's IndexedDB alongside the built-in HIPAA / Part 2 / PII presets. Re-select it any time to snap every column back to that plan — useful when you're experimenting with strategy changes and want a checkpoint to return to, or when the same dataset shape comes back every week.

  • Scope: per-browser, per-origin. Not synced across machines or browser profiles.
  • Contents: the column plan only (strategy + options for each column, per table).
  • Delete any time from the × button next to the saved preset name.

Exported profiles (portable JSON): use 'Export profile' in the Custom Rules toolbar to download a deidentify-profile.json file containing your full configuration — column plan, custom rules, foreign-key links, and the salt. 'Import profile' restores everything in one click.

  • Share with a teammate so everyone runs identical de-identification.
  • Commit to a repo as the team standard for a given source system.
  • Archive alongside a study protocol for reproducibility — same profile + same source data = byte-identical output.
  • Required for Study ID reproducibility: the exported salt is what makes the same SID come back next time.

Which to use? Saved presets for quick in-browser checkpoints and re-use. Exported profiles for sharing, archival, or anything that must survive a browser reset.

Preflight & Validation

What preflight checks

Before rewriting, DeIdentify scans your SQL for problems that would break referential mapping. Errors show line numbers.

  • COPY_UNSUPPORTED: pg_dump used COPY … FROM STDIN instead of INSERT. Re-export with pg_dump --inserts --column-inserts.
  • UNSUPPORTED_STATEMENT: forms like LOAD DATA INFILE, BULK INSERT, MERGE, INSERT … SELECT can't be rewritten row by row.
  • PK_NOT_IN_INSERT: an INSERT omits the primary key column. Referential mapping needs explicit PK values.
  • PK_NULL_VALUE / PK_DEFAULT_VALUE: primary keys were exported as NULL or DEFAULT (auto-increment). Re-export with explicit values.
  • NO_PRIMARY_KEY: table has no PRIMARY KEY declared — foreign key linking to it won't be consistent.
  • NO_SCHEMA: an INSERT references a table with no CREATE TABLE in the file — column-name detection still works but PK checks can't run.

Outputs & Reports

Download vs Copy

After a run, 'Download' saves the rewritten file with the filename you set in the 'save as' box. 'Copy' puts the same content on your clipboard — handy for pasting into a psql console or a scratch buffer.

Reading the Diff

The Diff tab in the preview pane shows original vs de-identified line by line. Red = removed (original), green = added (rewritten). Toggle 'Only changes' to collapse unchanged runs.

Run summary panel

After each run, the green summary panel shows how many columns and values were touched, which strategies ran, which detected types were involved, and which of your custom rules matched a column.

Audit report (.json / .csv)

The audit is a detailed log of what happened — per column, how many values were transformed, redacted, or kept, plus hashed samples so you can spot-check without exposing real PII. Keep this with the output file to prove your process.

File Converter (SQL ↔ CSV)

What the Converter does

The Converter page (/convert) turns a SQL dump into a CSV file, or a CSV file into a runnable SQL dump. Like the rest of DeIdentify, it runs entirely in your browser — no upload, no server round-trip — and is available in the offline single-file build.

  • SQL → CSV: parses CREATE TABLE + INSERT INTO statements and emits one RFC 4180-compliant .csv per table.
  • CSV → SQL: infers a schema from your CSV and emits a CREATE TABLE + batched INSERT INTO script in the dialect you pick.
  • The Converter does not de-identify. To scrub PII, run the output through the Home workbench (or convert first, then de-identify).
infoDefault direction is SQL → CSV. Use the toggle at the top of the page to switch to CSV → SQL. A 'Reset' button clears both input and output.

SQL → CSV (RFC 4180-compliant)

The CSV writer follows RFC 4180 so files open cleanly in Excel, Google Sheets, pandas, DuckDB, and every serious CSV parser.

  • Header row: always emitted, exactly matching the CREATE TABLE column order and count.
  • Text fields: always double-quoted so apostrophes and punctuation survive (e.g. O'Connor, "Smith, Jr.").
  • Internal double quotes are escaped by doubling (per spec): The "Big" Company → "The ""Big"" Company".
  • Embedded commas, CR, and LF inside a field are preserved — the field stays quoted and the row structure never breaks.
  • NULL values are written as an empty field (,,) — never the literal string 'NULL'.
  • Numbers and booleans are written unquoted when the column was inferred numeric/boolean.
  • Every row is normalized to the header's column count (missing trailing values become empty fields).

CSV → SQL (type inference + dialects)

When you paste or drop a CSV, DeIdentify scans the first 100 rows per column and picks the narrowest type that fits every sampled value. Priority order:

  • INTEGER — whole numbers only. Values with leading zeros (like '007') fall through to TEXT to preserve them.
  • DECIMAL / NUMERIC — numbers with a decimal point.
  • BOOLEAN — true/false, t/f, yes/no, 1/0 when the whole column matches.
  • DATE — recognizable ISO-style dates and common date/time formats.
  • TEXT — everything else (the safe fallback).

Primary key inference: a column named 'id' whose values are unique integers is auto-marked PRIMARY KEY. You can override this in the Schema Review panel.

Review & Refine Schema panel

After a CSV is parsed, an interactive schema panel appears between the input and the generated SQL. It shows each column with its guessed type as a colored badge and lets you override anything before the SQL is emitted.

  • Type dropdown per column — Integer, Decimal, Text, Date, Boolean.
  • Primary Key checkbox — only one column can be PK at a time (toggling another clears the previous).
  • Every change re-generates the SQL output immediately — no re-run button needed.
  • 'Skip ✕' hides the panel; 'Review & refine schema' brings it back.
tipA '💡 We automatically guessed…' indicator reminds you the schema is a best-effort inference. Always review before shipping the SQL into production.

SQL dialects supported

Pick the target dialect from the dropdown. Identifier quoting, data-type mapping, boolean encoding, and transaction syntax all change to match the target engine's conventions.

DialectIdentifier quoteINTEGERDECIMALBOOLEANDATETEXTTransaction
MySQL`backticks`INTDECIMAL(10,2)TINYINT(1)DATETEXTSTART TRANSACTION; … COMMIT;
Postgres"double quotes"INTEGERNUMERIC(10,2)BOOLEANDATETEXTBEGIN TRANSACTION; … COMMIT;
SQLite"double quotes"INTEGERREALINTEGER (0/1)TEXT (ISO)TEXTBEGIN TRANSACTION; … COMMIT;
ANSI"double quotes"INTEGERDECIMAL(10,2)BOOLEANDATEVARCHARBEGIN TRANSACTION; … COMMIT;
  • INSERT literals are dialect-aware: numbers unquoted, dates/text single-quoted with single-quote doubling, booleans encoded as TRUE/FALSE (Postgres/ANSI), 1/0 (MySQL TINYINT, SQLite), and NULL written as bare NULL.
  • Output is wrapped in a transaction block so a partial run rolls back cleanly.
  • INSERTs are batched for performance but stay within safe statement-size limits.

Identifier sanitizer & reserved words

CSV headers are cleaned into safe SQL identifiers automatically so the output runs without quoting gymnastics:

  • Lowercased. Spaces and special characters are replaced with underscores (First Name → first_name).
  • Leading digits get an underscore prefix (2024_totals → _2024_totals).
  • Reserved SQL keywords (SELECT, USER, ORDER, GROUP, TABLE, INDEX, …) get a '_field' suffix (user → user_field) so the DDL never collides.
  • The same rules apply to the inferred table name.

Limits & notes

  • The Converter is a structural transform — it does not scrub PII. Run the output through the Home workbench for de-identification.
  • In-memory mode: files ≤ 200 MB use the classic paste/preview flow with full type inference across every row.
  • Files > 200 MB automatically switch to streaming mode (see below) so multi-GB dumps convert without OOM.
  • The Converter is fully available in the offline single-file build — no network access required.

Streaming SQL → CSV (multi-GB files)

When you drop a .sql dump larger than 200 MB into the converter, DeIdentify swaps to a three-step streaming panel: scan → configure → stream-to-disk. Nothing ever sits in memory — the file is read in 4 MB slices, parsed by a Web Worker, and the CSV output is piped straight to a file on your disk via the File System Access API.

  • Step 1 — Scan: walks the whole file once to enumerate CREATE TABLE / INSERT INTO statements and count rows per table. No values are parsed, so a 7 GB dump typically scans in a minute or two.
  • Step 2 — Configure: pick the target table (one CSV per run), set delimiter / quote character / quote style / line ending / NULL rendering / include-header toggle.
  • Step 3 — Stream: pick a save location, then rows are read, converted, and written to disk in slices. Progress bar, throughput (MB/s and rows/s), elapsed time, and live ETA update every ~100 ms.
  • Cancel button is available during both the scan and the stream — clicking it stops the worker within a beat and aborts the disk write.
infoBrowser requirement: streaming saves use the File System Access API, which currently ships in Chrome, Edge, Opera, and Brave on desktop. Firefox and Safari can still use the in-memory converter for files ≤ 200 MB; for larger files use a Chromium-based browser.

Source SQL dialect selector — pick the engine your dump came from so identifier quoting and string escapes are parsed correctly:

DialectIdentifier quoteString escapesNotes
PostgreSQL (pg_dump)"double quotes"'' doubling + E'…' backslash escapesCOPY … FROM stdin blocks and $tag$…$tag$ dollar-quoted strings are skipped, not converted.
MySQL / MariaDB`backticks`'' doubling + \n \t \' \\ backslash escapes in every literalMulti-row INSERT INTO t VALUES (…), (…), (…) is supported.
SQLite (.dump)"double quotes"'' doubling only (no backslash escapes)Standard INSERT INTO output from sqlite3 .dump works out of the box.
Generic / ANSI SQL"double quotes"'' doubling onlySafe default for hand-written dumps or unknown sources.

Type & value handling during streaming:

  • No type inference is run in streaming mode. SQL values are decoded verbatim: NULL → empty CSV field (or your custom render), TRUE/FALSE → true/false, numerics unchanged, quoted strings unescaped, hex/bit literals (X'…', 0x…) passed through as their hex payload.
  • Column names come from CREATE TABLE first, then from an explicit INSERT column list, then fall back to col_1, col_2, … only if neither is present. The Step 2 panel warns you when a fallback is being used.
  • Rows are emitted RFC 4180-compliant: fields containing the delimiter, quote char, CR, or LF are wrapped in quotes; internal quotes are doubled. NULL is a bare empty field unless you set a custom NULL token.
  • Every row is written in order — no row is ever dropped. If a tuple has fewer values than the header, missing cells are emitted as empty fields; extras are truncated to the header width.
noteStreaming mode converts one table per run. If your dump has 20 tables and you need all of them as CSV, run the streamer 20 times (the scan result is cached between runs — just click 'Re-scan file' if you replace the source).

Downloading & Running Offline

One-click offline app

The fastest way to use DeIdentify air-gapped is the single-file build. Click the 'Download offline app' button in the green banner at the top of the page (or use the direct link below) — you get one HTML file with every asset inlined.

  • Save SqlDeidentify.html anywhere on your machine.
  • Double-click it. Any modern browser opens it as a local page (file://…).
  • Use it exactly like the web version. Drag-and-drop, presets, custom rules, all offline.
  • The bundle physically blocks fetch/XHR/WebSocket at runtime, so even if your machine is online, the tool can't transmit data.
infoSome browsers restrict features like IndexedDB when loading from file://. If your workspace (custom rules, salt) doesn't persist between sessions, serve the file with a local static server instead — see the next section.

Updating your offline copy (replacing the old file)

Every time the app is published, a new build is generated with a fresh version stamp (shown as v YYYY.MM.DD-HHMM in the top green banner and on the homepage offline card). To upgrade the copy on your machine:

  • Note the version currently shown in the green banner at the top of this site — that is the latest published build.
  • Open your existing local SqlDeidentify.html — the same version string appears in its own top banner. If it already matches, you are up to date.
  • If it is older, click 'Download offline app' in the banner (or the button on the homepage) to grab the new SqlDeidentify.html.
  • Save the download into the SAME folder as your old copy and let your browser overwrite the existing file (Chrome/Edge: 'Keep' → replaces; Firefox: check 'Replace'; Safari: it auto-renames — delete the old file, then rename the new one back to SqlDeidentify.html).
  • Close any browser tabs that still have the OLD file open, then double-click the new SqlDeidentify.html.
  • Your saved presets, custom rules, and salt are preserved automatically — they live in the browser's IndexedDB scoped to the file's origin, not inside the HTML itself.
infoKeeping the filename identical (SqlDeidentify.html) is what preserves your saved workspace. If you rename the new file or move it to a different folder, the browser treats it as a fresh origin and your custom presets will not appear (they are not lost — just scoped to the old path). Move the old file back or rename the new one to match.

Tip: bookmark this Knowledge Base page. The version badge in the top banner always reflects the newest published build, so you can check at a glance whether your local copy is behind.

Serve locally for persistent workspace

For a stable browser origin so IndexedDB persists your saved rules and plans across sessions:

# put SqlDeidentify.html in a folder, cd into it, then:
python3 -m http.server 8000
# or:  npx serve .
# or:  bunx serve .

# then open http://localhost:8000/SqlDeidentify.html

Build from source (advanced)

If you want to modify the tool or run the multi-file build:

git clone <your-project>
bun install
bun run build            # multi-file build in ./dist
bunx serve dist          # any static file server works

# Or rebuild the single-file offline bundle:
./scripts/build-standalone.sh
# → dist-standalone/SqlDeidentify-offline/SqlDeidentify.html

Verifying it's actually offline

  • Open your browser DevTools → Network tab, then load DeIdentify. You should see only local requests (or none, for the single-file build).
  • Turn off your Wi-Fi and load a real file. Everything should work identically.
  • Inspect the HTML: it declares Content-Security-Policy connect-src 'none', and installs a runtime kill-switch on fetch, XMLHttpRequest, and WebSocket.

Troubleshooting

A column was detected as the wrong type

Open the column table for that file, find the row, and change the Strategy or Type dropdown. Or add a custom rule so the detection is correct next time and for anyone who imports your profile.

Foreign keys aren't linking after de-identify

  • Check 'auto-link foreign keys' is on in the toolbar.
  • Confirm your INSERTs include the primary key column explicitly (preflight will flag if not).
  • Manually set the FK link on the column plan: set Strategy to 'Hash' and 'Link to' the parent table's PK column.

Preflight is showing red errors

Errors don't prevent you from running — but they mean referential mapping may be inconsistent. Read the line number and message, then either re-export your dump with the recommended flags or accept the risk if that table isn't linked to anything.

My file is very large

Everything runs in browser memory. Files up to a few hundred MB are usually fine on modern machines. If the tab hangs or crashes, split the dump (one table per file), de-identify each, and concatenate. For truly huge dumps, use the offline app on a workstation with more RAM.

Privacy & Compliance

Where does my data go?

Nowhere. Files you drop into DeIdentify are read with the browser's File API and processed in the same tab. There is no upload endpoint.

Your data lives only in the current browser tab's memory (in-memory JavaScript state). It is never written to IndexedDB, localStorage, sessionStorage, or any disk cache.

The de-identification engine runs entirely offline inside your browser. No network requests are made during parsing, analysis, or output generation. Even the offline build is configured to block any outgoing network calls while you work.

No service worker is registered, so there is no background cache of uploads, outputs, or audit reports. When you close or refresh the tab, everything in memory is released and gone.

tipThe chat assistant only sends your text question to the AI gateway. It never transmits file contents, analysis results, or output data. The chat assistant is not available offline.

This tool is not legal advice

noteDeIdentify helps you apply the mechanical steps of Safe Harbor and produce audit records, but compliance with HIPAA, GDPR, or any other regime is your responsibility. When in doubt, consult a privacy officer or qualified expert before sharing data.

Liability

By using this platform, you agree that 100% of liability for how you use it remains with you, not the platform. You are solely responsible for reviewing outputs, ensuring they meet your compliance obligations, and deciding whether a dataset is safe to share.

The platform provides automation and guidance, but it does not certify, guarantee, or warrant that any output satisfies legal or regulatory requirements.

Glossary

Glossary of terms

Plain-language definitions for every technical term used in the app and this knowledge base.

SQL
Structured Query Language. The standard language databases speak. A .sql file is just text containing commands like CREATE TABLE and INSERT INTO.
SQL dump
A text file exported from a database that recreates it exactly — the tables (CREATE TABLE) and every row of data (INSERT INTO ...).
CSV
Comma-Separated Values. A plain-text spreadsheet where each line is a row and columns are separated by commas.
Row
One record in a table — for example, one patient.
Column
One field on every row — for example, first_name or email.
Primary key (PK)
The column whose value uniquely identifies each row (usually 'id'). DeIdentify uses PK values to keep foreign-key links consistent after renaming.
Foreign key (FK)
A column that points to a row in another table. Example: encounters.patient_id points to patients.id. DeIdentify keeps these links intact.
PII
Personally Identifiable Information — any data that can identify a real person: names, emails, phone numbers, addresses, SSNs, etc.
PHI
Protected Health Information — PII combined with anything about a person's health, care, or payment for care. Governed by HIPAA in the US.
De-identification
Replacing PII/PHI with realistic-looking but fake values so the data is safe to share for testing, analytics, or research.
Pseudonymization
A form of de-identification where each real value is mapped to a stable fake one (Alice → Marta every time). Reversible only if the mapping is kept.
Anonymization
Stronger than pseudonymization — no mapping is kept and re-identification is meant to be infeasible.
HIPAA Safe Harbor
One of two HIPAA methods for de-identifying PHI. Removes 18 specific identifier categories (names, dates, geographic detail, IDs, etc.). The built-in preset applies all 18.
Expert Determination
The other HIPAA method — a statistician certifies re-identification risk is very small. DeIdentify supports this workflow with per-entity date shifts and generalization.
Salt
A short secret string mixed into hashes and fake generators so the same input maps to the same output within one run. Changing the salt gives you a totally new, still-consistent mapping.
Hash
A one-way function that turns a value into a fixed-length code. Deterministic (same input → same output) but not reversible without brute force.
Preflight
The pre-check DeIdentify runs before rewriting. Flags missing primary keys, INSERTs without column lists, and statement types the tool can't rewrite.
Preset
A saved bundle of column strategies (e.g. 'HIPAA Safe Harbor'). One click and every recognized column gets a sensible default.
Strategy
What to do with a column's values: Keep, Fake, Hash, Shift, Generalize, or Redact.
Date shift
Add or subtract a random-but-consistent number of days from every date. Preserves intervals (admission → discharge stays 4 days) while hiding the real calendar.
Per-entity shift
The same shift is applied to every date belonging to one patient. Different patients get different shifts. Recommended for HIPAA Expert Determination.
Per-table shift
The same shift is applied to every date in one table. Simpler; leaks less about individuals when a table has many people.
Generalize
Replace an exact value with a broader bucket. Example: '78 years old' → '75-84', '2024-03-18' → '2024', '10024' → '100XX'.
Redact / Null
Drop the value entirely — set it to NULL or an empty string. Use when the field is not needed downstream.
Referential integrity
The guarantee that foreign keys still line up after rewriting. When patients.id becomes '42→7891', encounters.patient_id also becomes '42→7891'.