feat: redact api keys/secrets/JWTs during session import (Privacy) #27

New Issue

2026-04-25T05:02:41+02:00

product-owner commented

2026-04-25 05:02:41 +02:00

Problem Statement

The session-search spec's "Security & Privacy" section (terraphim-agent-session-search-spec.md, lines 588-606) requires api key and secret redaction during session import:

api Keys: Redacted during import (regex patterns)
Secrets: Optional secret scanning with configurable patterns

Verification:

rg -l "redact|api_key|sk-|REDACTED|secret_pattern" crates/terraphim_sessions/   -> 0 matches
rg "redact" crates/terraphim-session-analyzer/src/                              -> 0 matches

Today every imported Session::messages[].content is stored verbatim, including any inlined OpenAI/Anthropic/AWS keys, GitHub tokens, JWTs, and .env paste-ins from coding sessions. This violates the privacy-first guarantee (Non-Goal 1: "privacy-first, local only") because:

The Tantivy index (issue #5) will surface the secrets via search.
/sessions export --format markdown will write them to disk in plaintext.
Robot mode JSON envelopes will return them to LLM agents.

We need a pluggable redaction stage at the connector boundary, applied uniformly across NativeClaudeConnector, ClaClaudeConnector, ClaCursorConnector, and any future connector (#3, #13, plus OpenCode/Codex tracked separately).

Acceptance Criteria

Add crates/terraphim_sessions/src/redaction.rs exporting Redactor, RedactionRule { name, pattern: regex::Regex, replacement }, RedactionReport { rule_hits: BTreeMap<String,usize> } with methods with_default_rules(), with_rules(...), redact(text) -> (String, RedactionReport), redact_session(&mut Session) -> RedactionReport
Default rules ship case-insensitive where relevant:
- OpenAI sk-[A-Za-z0-9]{20,} -> [REDACTED:openai-key]
- Anthropic sk-ant-[A-Za-z0-9_-]{20,} -> [REDACTED:anthropic-key]
- AWS access AKIA[0-9A-Z]{16} -> [REDACTED:aws-access-key]
- AWS secret (?i)aws_secret_access_key\s*=\s*[A-Za-z0-9/+=]{30,} -> [REDACTED:aws-secret]
- GitHub PAT gh[pousr]_[A-Za-z0-9]{36,} -> [REDACTED:github-pat]
- Bearer (?i)Bearer\s+[A-Za-z0-9._-]{20,} -> Bearer [REDACTED]
- JWT eyJ[A-Za-z0-9_-]+\.eyJ[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+ -> [REDACTED:jwt]
Sessionservice::import_with_options accepts redactor: Option<Arc<Redactor>>; default = Some(Redactor::with_default_rules()) -- OPT-OUT semantics
Each connector calls redactor.redact_session(&mut session) after building the Session and before caching
Aggregate RedactionReport returned from import() and surfaced in CLI: imported 42 sessions; redacted 3 OpenAI keys, 1 GitHub PAT
Robot envelope meta.redaction_report: { rule_hits: {..}, total: N }
Custom rules loadable from <terraphim_data_dir>/redaction.toml ([[rules]] name pattern replacement)
--no-redact flag on /sessions import (writes warning to stderr when used)
Apply redaction over Message::content AND CodeSnippet::content (code blocks are highest-leak surface)
Unit tests: each default rule fires exactly once on a fixture string; redact_session walks all messages including code snippets; counter increments correctly; idempotent (second pass yields empty report)
Integration test: import a fixture Claude JSONL containing a known fake sk-test-... and assert cached body equals [REDACTED:openai-key]
cargo test -p terraphim_sessions redaction passes
cargo test -p terraphim_sessions --features cla-full passes (existing tests still green)
cargo clippy -p terraphim_sessions -- -D warnings passes

Proposed Approach

New module crates/terraphim_sessions/src/redaction.rs (~200 LOC)
Reuse workspace regex dep; no new deps
Apply over message.content and CodeSnippet::content (highest-leak surface)
BTreeMap<String, usize> for stable JSON ordering
Wire into Sessionservice::import_* callsites (~5 places); redactor stays Option so tests can pass None
Surface report via existing import return type; extend with redaction: RedactionReport

Dependencies

None. Independent of #5 (Tantivy index), #3 (Cline), #13 (Generic MCP). Should land BEFORE #5 lands its writer so the index never sees raw secrets, but neither blocks the other.

Verification

cargo test -p terraphim_sessions redaction
cargo test -p terraphim_sessions --features cla-full
cargo clippy -p terraphim_sessions -- -D warnings
echo 'My key is sk-test-1234567890abcdefghij' > /tmp/fake-claude.jsonl
./target/debug/terraphim-agent --robot sessions import --source claude-code --path /tmp/fake-claude.jsonl | jq '.meta.redaction_report'

Expected: redaction_report.rule_hits["openai-key"] >= 1; cached session body equals [REDACTED:openai-key].

Scope

Single PR, ~350-450 LOC: redactor ~200, wiring across 3 connectors ~60, CLI/robot surfacing ~50, TOML loader ~30, tests ~120. Public-api change limited to ImportOptions and the new opt-out flag.

Upstream: terraphim-agent-session-search-spec.md "Security & Privacy" section (lines 588-606); privacy-first non-goal.

## Problem Statement The session-search spec's "Security & Privacy" section (terraphim-agent-session-search-spec.md, lines 588-606) requires api key and secret redaction during session import: > **api Keys**: Redacted during import (regex patterns) > **Secrets**: Optional secret scanning with configurable patterns Verification: ``` rg -l "redact|api_key|sk-|REDACTED|secret_pattern" crates/terraphim_sessions/ -> 0 matches rg "redact" crates/terraphim-session-analyzer/src/ -> 0 matches ``` Today every imported `Session::messages[].content` is stored verbatim, including any inlined OpenAI/Anthropic/AWS keys, GitHub tokens, JWTs, and `.env` paste-ins from coding sessions. This violates the privacy-first guarantee (Non-Goal 1: "privacy-first, local only") because: 1. The Tantivy index (issue #5) will surface the secrets via search. 2. `/sessions export --format markdown` will write them to disk in plaintext. 3. Robot mode JSON envelopes will return them to LLM agents. We need a pluggable redaction stage at the connector boundary, applied uniformly across `NativeClaudeConnector`, `ClaClaudeConnector`, `ClaCursorConnector`, and any future connector (#3, #13, plus OpenCode/Codex tracked separately). ## Acceptance Criteria - [ ] Add `crates/terraphim_sessions/src/redaction.rs` exporting `Redactor`, `RedactionRule { name, pattern: regex::Regex, replacement }`, `RedactionReport { rule_hits: BTreeMap<String,usize> }` with methods `with_default_rules()`, `with_rules(...)`, `redact(text) -> (String, RedactionReport)`, `redact_session(&mut Session) -> RedactionReport` - [ ] Default rules ship case-insensitive where relevant: - OpenAI `sk-[A-Za-z0-9]{20,}` -> `[REDACTED:openai-key]` - Anthropic `sk-ant-[A-Za-z0-9_-]{20,}` -> `[REDACTED:anthropic-key]` - AWS access `AKIA[0-9A-Z]{16}` -> `[REDACTED:aws-access-key]` - AWS secret `(?i)aws_secret_access_key\s*=\s*[A-Za-z0-9/+=]{30,}` -> `[REDACTED:aws-secret]` - GitHub PAT `gh[pousr]_[A-Za-z0-9]{36,}` -> `[REDACTED:github-pat]` - Bearer `(?i)Bearer\s+[A-Za-z0-9._-]{20,}` -> `Bearer [REDACTED]` - JWT `eyJ[A-Za-z0-9_-]+\.eyJ[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+` -> `[REDACTED:jwt]` - [ ] `Sessionservice::import_with_options` accepts `redactor: Option<Arc<Redactor>>`; default = `Some(Redactor::with_default_rules())` -- OPT-OUT semantics - [ ] Each connector calls `redactor.redact_session(&mut session)` after building the Session and before caching - [ ] Aggregate `RedactionReport` returned from `import()` and surfaced in CLI: `imported 42 sessions; redacted 3 OpenAI keys, 1 GitHub PAT` - [ ] Robot envelope `meta.redaction_report: { rule_hits: {..}, total: N }` - [ ] Custom rules loadable from `<terraphim_data_dir>/redaction.toml` (`[[rules]] name pattern replacement`) - [ ] `--no-redact` flag on `/sessions import` (writes warning to stderr when used) - [ ] Apply redaction over `Message::content` AND `CodeSnippet::content` (code blocks are highest-leak surface) - [ ] Unit tests: each default rule fires exactly once on a fixture string; `redact_session` walks all messages including code snippets; counter increments correctly; idempotent (second pass yields empty report) - [ ] Integration test: import a fixture Claude JSONL containing a known fake `sk-test-...` and assert cached body equals `[REDACTED:openai-key]` - [ ] `cargo test -p terraphim_sessions redaction` passes - [ ] `cargo test -p terraphim_sessions --features cla-full` passes (existing tests still green) - [ ] `cargo clippy -p terraphim_sessions -- -D warnings` passes ## Proposed Approach - New module `crates/terraphim_sessions/src/redaction.rs` (~200 LOC) - Reuse workspace `regex` dep; no new deps - Apply over `message.content` and `CodeSnippet::content` (highest-leak surface) - `BTreeMap<String, usize>` for stable JSON ordering - Wire into `Sessionservice::import_*` callsites (~5 places); redactor stays `Option` so tests can pass `None` - Surface report via existing import return type; extend with `redaction: RedactionReport` ## Dependencies None. Independent of #5 (Tantivy index), #3 (Cline), #13 (Generic MCP). Should land BEFORE #5 lands its writer so the index never sees raw secrets, but neither blocks the other. ## Verification ``` cargo test -p terraphim_sessions redaction cargo test -p terraphim_sessions --features cla-full cargo clippy -p terraphim_sessions -- -D warnings echo 'My key is sk-test-1234567890abcdefghij' > /tmp/fake-claude.jsonl ./target/debug/terraphim-agent --robot sessions import --source claude-code --path /tmp/fake-claude.jsonl | jq '.meta.redaction_report' ``` Expected: `redaction_report.rule_hits["openai-key"] >= 1`; cached session body equals `[REDACTED:openai-key]`. ## Scope Single PR, ~350-450 LOC: redactor ~200, wiring across 3 connectors ~60, CLI/robot surfacing ~50, TOML loader ~30, tests ~120. Public-api change limited to `ImportOptions` and the new opt-out flag. Upstream: terraphim-agent-session-search-spec.md "Security & Privacy" section (lines 588-606); privacy-first non-goal.

product-owner commented

2026-04-25 05:02:54 +02:00

Duplicate of #26 (created accidentally on retry). Closing.

product-owner closed this issue

2026-04-25 05:02:54 +02:00

root referenced this issue

2026-04-25 05:07:16 +02:00

feat: redact api keys/secrets/JWTs during session import (Privacy) #26

Sign in to join this conversation.