Everything K-culture — comebacks to K-beauty, straight to your inboxGet it in your inbox

METAL MEDIA

Redakto - The Incognito Tab for LLMs

arXiv:2608.182602026-08-20

An open-source tool that strips names and addresses before text hits an LLM, without hurting downstream accuracy

Redakto is an open-source tool that automatically detects and removes or replaces personally identifiable information (PII) in text before it is sent to a large language model (LLM). The authors tested it on emails, clinical notes and legal documents, and packaged it as a web app, REST APIs, and a Model Context Protocol (MCP) server that LLM assistants can call directly. Across two downstream tasks, redacted text performed about as well as the original, unredacted text.

METAL MEDIA explanatory visual

An open-source tool that strips names and addresses before text hits an LLM, without hurting downstream accuracy

  1. 01On German emails (CodE Alltag) and legal documents (LER), PII-detection accuracy (macro F1) reached about 0.95, while the smaller clinical corpus (GraSCCo) scored lower - a gap the authors attribute to limited training data rather than model architecture
  2. 02Three redaction strategies were compared: replacing entities with semantic placeholders, random masking, and generic masking; semantic placeholder masking preserved original performance best
  3. 03Utility was directly tested on two tasks: classifying a doctor's intent in medical dialogue turns, and predicting whether a human rights violation occurred in European Court of Human Rights case texts
  4. 04Even in documents packed with private entities (roughly 86 on average, sometimes several hundred), performance drops were mainly explained by document length, not by the redaction itself
  5. 05The system runs a Streamlit interface with a FastAPI backend, is containerized with Docker and Kubernetes, and both code and models are released openly
An explanatory diagram made by METAL MEDIA, not a figure supplied by the paper's authors.

What they did

  1. On German emails (CodE Alltag) and legal documents (LER), PII-detection accuracy (macro F1) reached about 0.95, while the smaller clinical corpus (GraSCCo) scored lower - a gap the authors attribute to limited training data rather than model architecture
  2. Three redaction strategies were compared: replacing entities with semantic placeholders, random masking, and generic masking; semantic placeholder masking preserved original performance best
  3. Utility was directly tested on two tasks: classifying a doctor's intent in medical dialogue turns, and predicting whether a human rights violation occurred in European Court of Human Rights case texts
  4. Even in documents packed with private entities (roughly 86 on average, sometimes several hundred), performance drops were mainly explained by document length, not by the redaction itself
  5. The system runs a Streamlit interface with a FastAPI backend, is containerized with Docker and Kubernetes, and both code and models are released openly
Figure 1: PII-detection performance on GraSCCo
Figure 1: PII-detection performance on GraSCCo
Table 1: PII-detection performance on CodE Alltag
ModelPrecRecF1
xlm-roberta-large0.9442 ±0.00.9358 ±0.00.9399 ±0.0
gelectra-large0.9433 ±0.00.9371 ±0.00.9401 ±0.0
bert-base-german-cased0.9251 ±0.00.9160 ±0.00.9204 ±0.0
Figure 2: Medical Intent classification on anonymized texts
Figure 2: Medical Intent classification on anonymized texts
Table 2: PII-detection performance on LER
ModelPrecRecF1
xlm-roberta-large0.940.950.94
gelectra-large0.950.960.95
bert-base-german-cased0.950.940.95
Figure 3: Entity density wise performance (Macro-F1) comparison for different redaction strategies (xlm-roberta-large)
Figure 3: Entity density wise performance (Macro-F1) comparison for different redaction strategies (xlm-roberta-large)
Table 3: Examples of redaction strategies applied to a clinical text
VersionText
Originalmiss edwards is here for evaluation of facial pain this is a 54-year-old female
Semantic Label Maskingmiss [PERSON] is here for evaluation of facial pain this is a [DATE] female
Random Maskingmiss lhyZXSX is here for evaluation of facial pain this is a vejE4fPRUxkG female
Generic Maskingmiss XXXX is here for evaluation of facial pain this is a XXXX female
Figure 4: Entity density wise performance (Macro-F1) comparison for different redaction strategies (bert-large-cased)
Figure 4: Entity density wise performance (Macro-F1) comparison for different redaction strategies (bert-large-cased)
Table 4: Binary Violation prediction on anonymized texts
Redaction Strategyxlm-roberta-largebert-large-casedelectra-large-discri.
No Redaction0.86 ±0.010.85 ±0.010.84 ±0.01
Semantic Label Masking0.86 ±0.010.84 ±0.010.82 ±0.02
Random Masking0.85 ±0.010.82 ±0.020.82 ±0.01
Generic Masking0.84 ±0.020.85 ±0.010.82 ±0.02
Figure 6: System design of Redakto
Figure 6: System design of Redakto

Why it matters

Uncertainty over EU privacy rules has been slowing adoption of LLMs in hospitals, legal offices and public administration, so empirical evidence that redaction doesn't hurt task quality directly addresses that barrier. Because it's open source and self-hostable, developers and researchers can adopt privacy-preserving LLM workflows without sending sensitive data to third-party services.

Terms in this paper

  • PII (personally identifiable information) · Details like names, addresses or birthdates that can identify a specific person
  • LLM (large language model) · An AI model trained on huge amounts of text to understand and generate language
  • pseudonymization · Replacing personal data with substitute identifiers instead of deleting it, keeping the text readable
  • Model Context Protocol (MCP) · A standard interface letting LLM assistants and agents call external tools directly
  • macro F1 · An accuracy measure averaged evenly across categories, so no single category dominates the score

Figures we cannot republish

  • Figure 5: Entity density wise performance (Macro-F1) comparison for different redaction strategies (electra-large-discriminator)
See the figures in the original paper →

Original abstract (English)

Large Language Models (LLMs) are being increasingly used in everyday applications. A major challenge in the context of LLMs or Artificial Intelligence (AI) in general is to ensure privacy when using them, meaning that personally identifiable information (PII) is removed from any text that enters an LLM. These challenges have become more urgent with novel EU legislation. Uncertainty around LLM usage with respect to privacy concerns in EU countries can be a major blocker for the speed of innovation and transfer from research to applications. Here we present \textbf{Redakto}, a tool that can be used for anonymizing text prior to feeding it to an LLM or other downstream text processing. We provide state-of-the-art functionalities for both redaction of PII but also when used for pseudonymization. These functionalities are exposed such that they can easily be used by end-users, through the Redakto web application, and by developers and researchers, via REST APIs and model context protocol (MCP) hooks. The implementation is fully open source, requires modest compute resources, and can be readily deployed on local hardware. In contrast to prior work and in order to better assess the quality of the anonymized texts, we conduct extensive empirical evaluations on textual data from legal and medical domain with respect to both privacy and utility of the redacted texts. Our empirical results demonstrate that the texts anonymized with different redaction strategies achieve utility scores on par with the original texts, suggesting that anonymization with Redakto can be used for LLM tasks without substantial negative impact for the tasks we explored.

Authors · Saurav Kumar Saha, Tom R\"ohr, Felix Bie{\ss}mann

Read on arXiv

Latest papers

All papers →

Latest from METAL MEDIA

Figures: Saurav Kumar Saha et al., arXiv:2608.18260, CC BY-SA 4.0