|Ben @ Grepture

How to Prevent Sensitive Data Leaks in LLM API Calls

Step-by-step guide to preventing PII, secrets, and sensitive data from leaking through LLM API calls — from audit to enforcement.

The problem nobody talks about

You're building a support AI feature. Your application assembles prompts from support tickets, user profiles, and order history. A typical request to OpenAI looks like this:

"Customer Sarah Miller (sarah@example.com, +1-555-0123) is asking about order #45231. Here's their account info: SSN ending 4532, address 742 Evergreen Terrace, Springfield..."

All of this flows to OpenAI's API. Every request. Every customer. Every time.

And it's not just customer support. Any AI feature that touches real data — code assistants pulling from private repos, analytics tools summarizing user behavior, content generators referencing internal docs — has the same problem.

What can leak:

  • PII (names, emails, phones, addresses) — GDPR/CCPA violation, potential fines in the millions
  • Secrets (API keys, database connection strings, OAuth tokens accidentally included in prompt context) — direct security breach
  • Proprietary data (source code, internal documentation, trade secrets) — IP theft risk
  • Regulated data (health information, financial records) — HIPAA, PCI-DSS, KYC/AML compliance violations

Most teams don't realize how much data is flowing until they look. The fix isn't "be more careful" — it's automated detection at the infrastructure level. This guide covers a step-by-step approach to finding, classifying, and controlling sensitive data in your LLM traffic using Grepture.

Step 1: Audit — understand what's being sent

Before you can protect anything, you need to see what's flowing through your AI pipeline. This is where most teams get their first surprise.

Set up Grepture with log-only rules — detection without blocking. Nothing changes about your application's behavior.

import Grepture from "@grepture/sdk";
import OpenAI from "openai";

const grepture = new Grepture({
  apiKey: process.env.GREPTURE_API_KEY,
  proxyUrl: "https://proxy.grepture.com",
});

const openai = new OpenAI({
  ...grepture.clientOptions(),
  apiKey: process.env.OPENAI_API_KEY,
});

// That's it. All requests now flow through Grepture.
// Configure detection rules in the dashboard — start with log-only mode.

Three lines of code. Now head to the dashboard and enable detection:

  • Regex detection for structured PII — emails, phone numbers, SSNs, credit card numbers, API keys, bearer tokens. 50+ built-in patterns on the free tier.
  • AI detection for names and locations — these appear in natural language and slip past regex. Enable local AI models to catch them.

Set all rules to log-only mode with severity levels (info, warn, or critical) so you can see what's happening without affecting production traffic.

Let it run for a week against your production traffic. Then review the dashboard traffic log — you'll see every request, what was detected, and the severity.

Teams consistently underestimate how much PII flows through their AI pipelines. We've seen applications sending full customer profiles, including social security numbers, to LLMs on every request — and nobody knew until they looked. For more on what to look for, see our PII detection best practices guide.

Step 2: Detection rules — structured data + AI

After the audit phase, you know what's flowing. Now set up proper detection rules. Two layers that work together.

Regex rules for structured data. Fast, deterministic, predictable. 50+ built-in patterns (80+ on Pro) covering:

  • Email addresses
  • Phone numbers (international formats)
  • Social security numbers
  • Credit card numbers (with Luhn validation)
  • API keys and bearer tokens
  • Database connection strings
  • IP addresses
  • AWS access keys, GitHub tokens, Stripe keys, and more

Each rule can be configured with a specific action (log, redact, mask, block) and severity level. Start with logging, then tighten.

AI detection for unstructured data. Regex can't catch "Sarah Miller lives in Springfield" — no pattern to match. Local AI models fill the gap:

  • Names — person names in any format, any language
  • Locations — cities, addresses, countries mentioned in natural language
  • Organizations — company names, institution references

All AI models run locally on Grepture infrastructure. Your data doesn't get forwarded anywhere else.

Prompt injection as a separate concern. While you're setting up detection, enable prompt injection scoring too. A different threat than data leakage, but equally important when processing user-generated content. See our prompt injection prevention guide for a deep dive.

Configure conditions to target specific endpoints, models, or request patterns. Set sampling rates — 100% for critical regex rules. If you're on the free tier (25 AI detection requests/month), use sampling for AI-powered detection while you evaluate, then upgrade to Pro for unlimited.

Step 3: Mask and restore — for user-facing AI features

For features where the LLM response goes back to the user — chatbots, support agents, content generation — the data needs to survive the round trip. Permanently redacting a customer's name from a support response makes it useless.

How mask-and-restore works:

  1. Grepture detects PII in the outgoing request
  2. Sensitive values get replaced with tokens (e.g., Sarah Miller becomes [NAME_1])
  3. Original values are stored in a secure vault with a configurable TTL
  4. The LLM processes the tokenized prompt and generates a response using the same tokens
  5. Grepture intercepts the response and restores the original values

The user gets a natural response with their real data. The LLM never sees it.

Configure mask-and-restore rules in the dashboard for the data types that matter:

  • Names — mask with AI detection, restore in response
  • Email addresses — mask with regex, restore in response
  • Phone numbers — mask with regex, restore in response
  • Addresses — mask with AI detection, restore in response

Works with streaming too. Grepture handles SSE natively, detokenizing streamed chunks in real time.

For a deep dive into how it works under the hood — vault TTL configuration, edge cases — see our mask and restore guide.

Step 4: Handle secrets differently — block, don't mask

Secrets are not PII. They need a different response.

If an API key, database connection string, or OAuth token appears in a prompt, something is wrong. Masking a secret and sending it to the LLM hides the symptom — the real problem is that credentials are ending up in prompt context.

Set up block rules for secrets:

  • If an API key or connection string is detected, reject the request with a clear error message
  • Log the incident at critical severity so your team can investigate and fix the root cause
  • The blocked request never reaches the LLM provider

For secrets in less critical contexts — example API keys in documentation content — use permanent redaction instead of mask-and-restore. Replace with [REDACTED] and don't store the original. No reason to round-trip a secret through an LLM.

Rule of thumb:

  • Mask-and-restore for PII that's part of normal operation (names, emails, phones in user-facing features)
  • Block for secrets that indicate a bug or misconfiguration (API keys, tokens, connection strings)
  • Permanent redaction for sensitive data that doesn't need to survive the round trip (SSNs, credit card numbers in analytical queries)

Step 5: Monitor and tighten

Detection rules aren't set-and-forget. Your application evolves, new data sources get connected, new AI features get shipped. Ongoing monitoring turns initial protection into durable security.

Review the dashboard regularly. Check what's being detected, look for false positives, and identify patterns you haven't accounted for.

Tune thresholds. If you're getting false positives on common names that aren't PII in your context, raise the confidence threshold. Missing edge cases? Lower it.

Add custom patterns for your specific data:

  • Internal user IDs or account numbers with predictable formats
  • Custom token formats used by your authentication system
  • Proprietary data markers or classification labels
  • Industry-specific identifiers (policy numbers, patient IDs, case references)

Enable zero-data mode (Business+) for maximum data minimization. Every request processed in memory only. No bodies, no headers touch storage. Only operational metadata — method, status code, latency, which rules fired — gets logged.

Generate compliance reports (Business+) for audit evidence. What rules are active, what data was found, what actions were taken — what your compliance team needs for GDPR, HIPAA, or SOC 2 audits.

New AI features are automatically protected. Every request flows through the proxy.

Putting it all together

The full pipeline, from zero to protected:

  1. SDK integration — one-time setup, 3 lines of code. See the quickstart guide.
  2. Audit phase — log-only mode, 1 week of production traffic. Understand what's flowing.
  3. PII detection — regex + AI rules, mask-and-restore for user-facing features. See PII detection best practices.
  4. Secret detection — block rules for API keys, tokens, and connection strings.
  5. Prompt injection detection — AI scoring, start with logging. See prompt injection prevention.
  6. Ongoing monitoring — dashboard review, threshold tuning, custom patterns.

The setup takes under an hour. The audit phase is where you invest time — and where you'll learn the most about what your AI features are doing with your data.

This satisfies GDPR data minimization requirements, EU AI Act robustness requirements, and gives your compliance team the audit trail they need. For details on how these regulations apply to AI systems, see our EU AI Act compliance guide.

For a broader view of the tool landscape, check our LLM security tools comparison. For rule configuration details, see the configuration docs.

The data flowing through your AI pipeline right now won't protect itself. Set up the audit, see what's there, and start closing the gaps.