PII and Data Privacy
Engineering personal-data protection in LLM apps: where PII flows (prompts, logs, embeddings, caches), detection and redaction with Presidio, zero data retention, training opt-outs, data residency, and GDPR implications.
Contents
PII and data privacy engineering for LLM applications means controlling where personal data goes as it passes through prompts, retrieval, model providers, logs and training sets. In practice that means four things: minimise or transform personal data before it leaves your boundary (detection and redaction), choose provider settings that limit what is stored and where (zero data retention, training opt-outs, data residency), keep the stores you own (traces, vector indexes, caches) inside your retention and erasure processes, and be able to show a regulator a lawful basis for each step. OWASP ranks Sensitive Information Disclosure as LLM02 in both the 2025 and 2026 editions, and notes that system-prompt instructions alone cannot prevent it, because prompt injection can override them (OWASP LLM02, OWASP 2026).
Where personal data lives in an LLM system
flowchart LR
U["User input<br/>(free text, files)"] --> R["Redaction / pseudonymisation<br/>(Presidio, guardrail)"]
R --> C["Context assembly<br/>(RAG chunks, memory, tool results)"]
V[("Vector store<br/>embeddings + chunk text")] --> C
C --> P["Model provider<br/>(abuse logs, caches, stored state)"]
P --> O["Output filter / re-identification"]
O --> U2["User"]
C -.-> L[("Traces & logs<br/>(observability)")]
P -.-> L
L -.-> FT[("Eval / fine-tuning datasets")]Synthesis: each box in the diagram is a separate retention and erasure problem. The ones teams most often forget:
- Provider-side stores: abuse-monitoring logs, stored responses, files, prompt caches. OpenAI keeps prompt-cache tensors in GPU-local storage for up to 24 hours (OpenAI). Vertex implicit caches are deleted within 24 hours (Vertex).
- Your traces. Observability tools capture full prompts and completions by default.
- Embeddings. Vec2Text recovers 92% of 32-token inputs exactly from their embeddings, and full names from clinical-note embeddings (Morris et al.). Synthesis: treat a vector index as containing the source text, with the same access control and erasure duties. See embeddings, vector databases.
- Model weights, if you fine-tune on personal data. Models memorise training data verbatim, even sequences seen in a single document, and larger models memorise more (Carlini et al.). NIST lists memorisation, leakage and inference of undisclosed sensitive attributes under its data-privacy risk (NIST AI 600-1).
Detection and redaction with Presidio
Presidio is the standard open-source building block. It was started by Microsoft and is now community-owned under “Data Privacy Stack” (Presidio).
- Analyzer. An
AnalyzerEngineruns every recognizer in aRecognizerRegistryover text pre-processed by an NLP engine (spaCy by default; Stanza or Hugging Face Transformers optional). Recognizers combine NER, regular expressions, checksum validation and rule logic. A context enhancer raises confidence when nearby words, matched by lemma or phrase, support the entity. Each result is an entity type, a span and a score. Recognizers carry a country code, so you can scope detection (e.g. EU-only), and you can add custom recognizers (Presidio). - Anonymizer operators.
replace(default<ENTITY_TYPE>),redact,mask(chars_to_mask,masking_char,from_end),hash(salted SHA-256/512; reuse one salt to get consistent hashes),encrypt(keyed),custom,surrogate_ahds(realistic surrogates via Azure Health Data Services),keep. Onlyencryptis reversible, viadecryptwith the same key (Presidio). - Also: an image redactor (OCR, then redaction) and a structured-data mode for tables (Presidio).
| Operator | Reversible | LLM still understands? | Use when |
|---|---|---|---|
replace with type tag | no | partly: it knows “a PERSON” | analytics, logs |
hash (fixed salt) | no (linkable) | consistent tokens across a conversation | joining records without exposing values |
encrypt | yes, with key | consistent opaque tokens | round-trip pseudonymisation: redact before the provider, decrypt in the output |
mask | no | partial (e.g. last 4 digits) | UI display, support tickets |
| surrogate | no | yes, it reads naturally | test data and fine-tuning sets |
Synthesis on the round-trip pattern: replace each entity with a consistent token before the API call, keep the token-to-value map (or key) inside your boundary, and restore values in the response. The provider then sees pseudonymised data. Under GDPR that is still personal data for you, because you hold the key (GDPR Art. 4), but it shrinks what a provider breach or log can expose.
Managed alternatives exist. Bedrock Guardrails’ sensitive-information filters block or mask PII and custom regexes on inputs and outputs, and do not evaluate reasoning blocks (Bedrock Guardrails). See guardrail tooling.
Limits. Presidio’s docs warn that automated detection cannot guarantee it finds all sensitive data (Presidio). Synthesis: expect misses on names that are also ordinary words, non-English text, IDs split across tokens or lines, and quasi-identifiers (job title + town + date) that no single recognizer flags. Measure recall on your own labelled sample before relying on it, and red-team leakage with promptfoo’s pii:direct, pii:session, pii:social, pii:api-db and cross-session-leak plugins (promptfoo plugins; see LLM red teaming).
Provider controls: training, retention, residency (as of 2026-11)
| Provider | Training on API data | Default retention | Zero-retention option | Residency |
|---|---|---|---|---|
| Anthropic API | Commercial products excluded from the consumer training choice | deleted within 30 days; flagged content up to 2 years | ZDR per organization; no browser (CORS) calls; Batch, Files and code execution step outside it; “Covered Models” need 30-day retention | inference_geo global/us per request (US-only at 1.1x price, Claude 4.6+); workspace geo us only, immutable |
| OpenAI API | not since 2023-03-01 unless opted in | abuse logs up to 30 days; stored responses ≥30 days | Modified Abuse Monitoring or ZDR (forces store=false); not for files, vector stores, batch, fine-tuning, assistants | regional domains; US, EU, UAE (and some models in AU/JP/IN) process in-region, CA/SG/KR/UK store only; +10% on newer models |
| Amazon Bedrock | providers have no access to prompts | per model DataRetentionMode | mode none, enforceable by SCP; some models require aws_review | per Region; cross-Region inference stores in the destination Region |
| Vertex AI | not without permission | caches ≤24 h; grounding logs (Search: 3 days, Maps: 30 days) | disable caching and logging; use Web Grounding for Enterprise | regional and multi-region endpoints |
| Azure / Foundry | not used for training; not shared with OpenAI | abuse monitoring store in resource geography | Modified abuse monitoring (ContentLogging=false) | Standard (in geography), DataZone, Global |
Sources: Anthropic retention, Anthropic residency, Anthropic consumer terms, OpenAI data controls, OpenAI residency, Bedrock retention, Bedrock data protection, Vertex, Foundry.
Traps hidden in that table:
store=falseis not ZDR. On Bedrock only modenoneguarantees zero retention (Bedrock). On Vertex partner OpenAI models,store: falsealone doesn’t enable ZDR (Vertex).- Stateful features break ZDR. Anthropic’s API does not block ineligible features under ZDR; using one simply takes that data outside ZDR (Anthropic). OpenAI files and vector stores are ineligible (OpenAI).
- Flagged content is kept anyway. Trust-and-safety holds override ZDR at Anthropic (up to 2 years) and OpenAI (Anthropic, OpenAI).
- Storage residency is not inference residency. OpenAI’s storage-only regions may process outside the region (OpenAI residency). Anthropic’s
inference_geodoes not apply on Bedrock, Google-managed Vertex or Foundry, where the endpoint or deployment type decides (it does work on Claude Platform on AWS and Anthropic-managed Claude on Google Cloud) (Anthropic residency). - Consumer is not commercial. Consumer Claude plans let users opt in to training, with 5-year retention if they do; API and cloud-platform use is excluded (Anthropic consumer terms). Synthesis: employees pasting customer data into consumer chat apps is a governance problem, not an API setting.
See enterprise LLM deployment, Amazon Bedrock, Google Vertex AI, Azure AI Foundry.
GDPR implications for prompts, logs, embeddings and models
Prompts and logs that mention a person are personal data. Sending them to an API is processing (GDPR Art. 4(2)). Synthesis: an API provider acting on your instructions under a data processing agreement is normally a processor in the Art. 4(8) sense; check its terms, since any use for its own purposes (e.g. consumer-style training) changes that role. Pseudonymised data stays personal data (GDPR Art. 4(5)).
Anonymity is relative and must be demonstrated. The EDPB’s draft 2026 anonymisation guidelines use three criteria (no isolation, no linkage, no inference). They state that meaningful inferences drawn by prompting a model, and synthetic data, can fail the no-inference test, so outputs are not anonymous by default (EDPB 02/2026).
Are model weights personal data? Hamburg’s DPA argued in 2024 that LLMs store no personal data, so data-subject rights attach to the system’s inputs and outputs (Hamburg). The EDPB’s Opinion 28/2024 is stricter: a model trained on personal data is anonymous only if both extraction and query-based disclosure are insignificant, judged case by case (EDPB 28/2024).
Contradiction: Hamburg says training-phase violations don’t affect the lawfulness of deploying the model. The EDPB says it depends: a deployer must assess how a third-party model was developed, unless the model was effectively anonymised. Prefer the EDPB position, which is the EU-wide one.
Legal basis. Legitimate interest needs the three-step test (legitimate interest, necessity with data minimisation, balancing against data subjects’ reasonable expectations) (EDPB 28/2024).
Erasure and access. Synthesis: a deletion request must reach every copy: traces, conversation memory (agent memory), vector-index entries and their chunk text, eval datasets, provider-stored state (responses, files, threads), and any fine-tuned adapter. Weights cannot be surgically edited, which is a strong argument for keeping personal data in retrieval, where it can be deleted, rather than in fine-tuning. Design document IDs so one data subject’s chunks can be found and deleted.
See AI governance and compliance for the EU AI Act layer.
Practical guidance
Synthesis, built from the sources above:
- Minimise first. Don’t retrieve or send fields the task doesn’t need, and scope retrieval to the requesting user (Notebook: web/AI security, OWASP LLM02).
- Redact at the boundary when the provider or the logs shouldn’t see raw identifiers. Use reversible
encrypttokens when the answer must name the person, and type tags for analytics. Skip redaction when the task is about the identifiers (e.g. KYC extraction). In that case, rely on contractual and technical controls: ZDR, residency, a DPA or BAA. - Turn on provider controls deliberately. Request ZDR or MAM; avoid stateful endpoints, or accept their retention; pin the inference region; enforce at the org level (Bedrock SCPs, Anthropic
allowed_inference_geos). - Treat your own stores as in scope. Redact or hash in traces, set TTLs, and encrypt vector stores with access control equal to the source documents.
- Test leakage like any other failure. Run cross-user and cross-session leakage, system-prompt extraction and PII elicitation in your red-team suite, and add PII checks to guardrails on output.
Sources
- Presidio documentation
- OWASP LLM02 Sensitive Information Disclosure · OWASP LLM Top 10 2026
- Carlini et al. 2021 · Morris et al. 2023, Vec2Text
- Anthropic API and data retention · Anthropic data residency · Anthropic consumer terms update
- OpenAI data controls · OpenAI data residency
- Bedrock data retention · Bedrock data protection · Bedrock Guardrails
- Vertex AI data governance · Microsoft Foundry data privacy
- GDPR Article 4 · EDPB Opinion 28/2024 · EDPB Guidelines 02/2026 · Hamburg DPA discussion paper
- Promptfoo red team plugins · NIST AI 600-1 · Notebook: Web, API & AI Security