PII and Data Privacy

Engineering personal-data protection in LLM apps: where PII flows (prompts, logs, embeddings, caches), detection and redaction with Presidio, zero data retention, training opt-outs, data residency, and GDPR implications.

Contents

PII and data privacy engineering for LLM applications means controlling where personal data goes as it passes through prompts, retrieval, model providers, logs and training sets. In practice that means four things: minimise or transform personal data before it leaves your boundary (detection and redaction), choose provider settings that limit what is stored and where (zero data retention, training opt-outs, data residency), keep the stores you own (traces, vector indexes, caches) inside your retention and erasure processes, and be able to show a regulator a lawful basis for each step. OWASP ranks Sensitive Information Disclosure as LLM02 in both the 2025 and 2026 editions, and notes that system-prompt instructions alone cannot prevent it, because prompt injection can override them (OWASP LLM02, OWASP 2026).

Where personal data lives in an LLM system

flowchart LR
  U["User input<br/>(free text, files)"] --> R["Redaction / pseudonymisation<br/>(Presidio, guardrail)"]
  R --> C["Context assembly<br/>(RAG chunks, memory, tool results)"]
  V[("Vector store<br/>embeddings + chunk text")] --> C
  C --> P["Model provider<br/>(abuse logs, caches, stored state)"]
  P --> O["Output filter / re-identification"]
  O --> U2["User"]
  C -.-> L[("Traces & logs<br/>(observability)")]
  P -.-> L
  L -.-> FT[("Eval / fine-tuning datasets")]

Synthesis: each box in the diagram is a separate retention and erasure problem. The ones teams most often forget:

  • Provider-side stores: abuse-monitoring logs, stored responses, files, prompt caches. OpenAI keeps prompt-cache tensors in GPU-local storage for up to 24 hours (OpenAI). Vertex implicit caches are deleted within 24 hours (Vertex).
  • Your traces. Observability tools capture full prompts and completions by default.
  • Embeddings. Vec2Text recovers 92% of 32-token inputs exactly from their embeddings, and full names from clinical-note embeddings (Morris et al.). Synthesis: treat a vector index as containing the source text, with the same access control and erasure duties. See embeddings, vector databases.
  • Model weights, if you fine-tune on personal data. Models memorise training data verbatim, even sequences seen in a single document, and larger models memorise more (Carlini et al.). NIST lists memorisation, leakage and inference of undisclosed sensitive attributes under its data-privacy risk (NIST AI 600-1).

Detection and redaction with Presidio

Presidio is the standard open-source building block. It was started by Microsoft and is now community-owned under “Data Privacy Stack” (Presidio).

  • Analyzer. An AnalyzerEngine runs every recognizer in a RecognizerRegistry over text pre-processed by an NLP engine (spaCy by default; Stanza or Hugging Face Transformers optional). Recognizers combine NER, regular expressions, checksum validation and rule logic. A context enhancer raises confidence when nearby words, matched by lemma or phrase, support the entity. Each result is an entity type, a span and a score. Recognizers carry a country code, so you can scope detection (e.g. EU-only), and you can add custom recognizers (Presidio).
  • Anonymizer operators. replace (default <ENTITY_TYPE>), redact, mask (chars_to_mask, masking_char, from_end), hash (salted SHA-256/512; reuse one salt to get consistent hashes), encrypt (keyed), custom, surrogate_ahds (realistic surrogates via Azure Health Data Services), keep. Only encrypt is reversible, via decrypt with the same key (Presidio).
  • Also: an image redactor (OCR, then redaction) and a structured-data mode for tables (Presidio).
OperatorReversibleLLM still understands?Use when
replace with type tagnopartly: it knows “a PERSON”analytics, logs
hash (fixed salt)no (linkable)consistent tokens across a conversationjoining records without exposing values
encryptyes, with keyconsistent opaque tokensround-trip pseudonymisation: redact before the provider, decrypt in the output
masknopartial (e.g. last 4 digits)UI display, support tickets
surrogatenoyes, it reads naturallytest data and fine-tuning sets

Synthesis on the round-trip pattern: replace each entity with a consistent token before the API call, keep the token-to-value map (or key) inside your boundary, and restore values in the response. The provider then sees pseudonymised data. Under GDPR that is still personal data for you, because you hold the key (GDPR Art. 4), but it shrinks what a provider breach or log can expose.

Managed alternatives exist. Bedrock Guardrails’ sensitive-information filters block or mask PII and custom regexes on inputs and outputs, and do not evaluate reasoning blocks (Bedrock Guardrails). See guardrail tooling.

Limits. Presidio’s docs warn that automated detection cannot guarantee it finds all sensitive data (Presidio). Synthesis: expect misses on names that are also ordinary words, non-English text, IDs split across tokens or lines, and quasi-identifiers (job title + town + date) that no single recognizer flags. Measure recall on your own labelled sample before relying on it, and red-team leakage with promptfoo’s pii:direct, pii:session, pii:social, pii:api-db and cross-session-leak plugins (promptfoo plugins; see LLM red teaming).

Provider controls: training, retention, residency (as of 2026-11)

ProviderTraining on API dataDefault retentionZero-retention optionResidency
Anthropic APICommercial products excluded from the consumer training choicedeleted within 30 days; flagged content up to 2 yearsZDR per organization; no browser (CORS) calls; Batch, Files and code execution step outside it; “Covered Models” need 30-day retentioninference_geo global/us per request (US-only at 1.1x price, Claude 4.6+); workspace geo us only, immutable
OpenAI APInot since 2023-03-01 unless opted inabuse logs up to 30 days; stored responses ≥30 daysModified Abuse Monitoring or ZDR (forces store=false); not for files, vector stores, batch, fine-tuning, assistantsregional domains; US, EU, UAE (and some models in AU/JP/IN) process in-region, CA/SG/KR/UK store only; +10% on newer models
Amazon Bedrockproviders have no access to promptsper model DataRetentionModemode none, enforceable by SCP; some models require aws_reviewper Region; cross-Region inference stores in the destination Region
Vertex AInot without permissioncaches ≤24 h; grounding logs (Search: 3 days, Maps: 30 days)disable caching and logging; use Web Grounding for Enterpriseregional and multi-region endpoints
Azure / Foundrynot used for training; not shared with OpenAIabuse monitoring store in resource geographyModified abuse monitoring (ContentLogging=false)Standard (in geography), DataZone, Global

Sources: Anthropic retention, Anthropic residency, Anthropic consumer terms, OpenAI data controls, OpenAI residency, Bedrock retention, Bedrock data protection, Vertex, Foundry.

Traps hidden in that table:

  • store=false is not ZDR. On Bedrock only mode none guarantees zero retention (Bedrock). On Vertex partner OpenAI models, store: false alone doesn’t enable ZDR (Vertex).
  • Stateful features break ZDR. Anthropic’s API does not block ineligible features under ZDR; using one simply takes that data outside ZDR (Anthropic). OpenAI files and vector stores are ineligible (OpenAI).
  • Flagged content is kept anyway. Trust-and-safety holds override ZDR at Anthropic (up to 2 years) and OpenAI (Anthropic, OpenAI).
  • Storage residency is not inference residency. OpenAI’s storage-only regions may process outside the region (OpenAI residency). Anthropic’s inference_geo does not apply on Bedrock, Google-managed Vertex or Foundry, where the endpoint or deployment type decides (it does work on Claude Platform on AWS and Anthropic-managed Claude on Google Cloud) (Anthropic residency).
  • Consumer is not commercial. Consumer Claude plans let users opt in to training, with 5-year retention if they do; API and cloud-platform use is excluded (Anthropic consumer terms). Synthesis: employees pasting customer data into consumer chat apps is a governance problem, not an API setting.

See enterprise LLM deployment, Amazon Bedrock, Google Vertex AI, Azure AI Foundry.

GDPR implications for prompts, logs, embeddings and models

  • Prompts and logs that mention a person are personal data. Sending them to an API is processing (GDPR Art. 4(2)). Synthesis: an API provider acting on your instructions under a data processing agreement is normally a processor in the Art. 4(8) sense; check its terms, since any use for its own purposes (e.g. consumer-style training) changes that role. Pseudonymised data stays personal data (GDPR Art. 4(5)).

  • Anonymity is relative and must be demonstrated. The EDPB’s draft 2026 anonymisation guidelines use three criteria (no isolation, no linkage, no inference). They state that meaningful inferences drawn by prompting a model, and synthetic data, can fail the no-inference test, so outputs are not anonymous by default (EDPB 02/2026).

  • Are model weights personal data? Hamburg’s DPA argued in 2024 that LLMs store no personal data, so data-subject rights attach to the system’s inputs and outputs (Hamburg). The EDPB’s Opinion 28/2024 is stricter: a model trained on personal data is anonymous only if both extraction and query-based disclosure are insignificant, judged case by case (EDPB 28/2024).

    Contradiction: Hamburg says training-phase violations don’t affect the lawfulness of deploying the model. The EDPB says it depends: a deployer must assess how a third-party model was developed, unless the model was effectively anonymised. Prefer the EDPB position, which is the EU-wide one.

  • Legal basis. Legitimate interest needs the three-step test (legitimate interest, necessity with data minimisation, balancing against data subjects’ reasonable expectations) (EDPB 28/2024).

  • Erasure and access. Synthesis: a deletion request must reach every copy: traces, conversation memory (agent memory), vector-index entries and their chunk text, eval datasets, provider-stored state (responses, files, threads), and any fine-tuned adapter. Weights cannot be surgically edited, which is a strong argument for keeping personal data in retrieval, where it can be deleted, rather than in fine-tuning. Design document IDs so one data subject’s chunks can be found and deleted.

See AI governance and compliance for the EU AI Act layer.

Practical guidance

Synthesis, built from the sources above:

  1. Minimise first. Don’t retrieve or send fields the task doesn’t need, and scope retrieval to the requesting user (Notebook: web/AI security, OWASP LLM02).
  2. Redact at the boundary when the provider or the logs shouldn’t see raw identifiers. Use reversible encrypt tokens when the answer must name the person, and type tags for analytics. Skip redaction when the task is about the identifiers (e.g. KYC extraction). In that case, rely on contractual and technical controls: ZDR, residency, a DPA or BAA.
  3. Turn on provider controls deliberately. Request ZDR or MAM; avoid stateful endpoints, or accept their retention; pin the inference region; enforce at the org level (Bedrock SCPs, Anthropic allowed_inference_geos).
  4. Treat your own stores as in scope. Redact or hash in traces, set TTLs, and encrypt vector stores with access control equal to the source documents.
  5. Test leakage like any other failure. Run cross-user and cross-session leakage, system-prompt extraction and PII elicitation in your red-team suite, and add PII checks to guardrails on output.

Sources

Suggest changes →