feat: Implement Tantivy-based session index for full-text search #5

New Issue

2026-04-22T09:42:08+02:00

product-owner commented

2026-04-22 09:42:08 +02:00

Problem Statement

Session search currently uses naive in-memory string matching (String::contains) which is O(n) over all messages and does not support relevance ranking, phrase queries, or fuzzy matching. The terraphim_sessions/src/index/ directory exists but is empty.

Current State

Search implementation: crates/terraphim_sessions/src/service.rs:181 (linear scan)
Empty index module: crates/terraphim_sessions/src/index/ (no files)
No Tantivy dependency in terraphim_sessions/Cargo.toml
Search returns all matches without ranking

Acceptance Criteria

Add tantivy dependency to terraphim_sessions/Cargo.toml
Define Tantivy index schema with session/message fields
Implement SessionIndex with add/search methods
Integrate index into SessionService for automatic indexing on import
Search returns ranked results by relevance
Support phrase queries and prefix matching
Index persists to disk between sessions
cargo test -p terraphim_sessions passes
cargo clippy -- -D warnings passes

Proposed Approach

Add tantivy = "0.21" to terraphim_sessions/Cargo.toml
Create crates/terraphim_sessions/src/index/mod.rs with:
- SessionIndexSchema defining fields (session_id, message_id, content, code_content, timestamp, role, source)
- SessionIndex struct with add_session() and search() methods
- Custom tokenizers for code (edge n-gram) and natural language
Modify SessionService to maintain a SessionIndex alongside the cache
Update search() to use Tantivy when index is available, fallback to linear scan

Dependencies

None

Verification

cargo test -p terraphim_sessions
cargo build -p terraphim_agent --features repl-sessions
cargo clippy -p terraphim_sessions -- -D warnings

Scope

New module (~300 lines) + modifications to service.rs. Single PR.

Upstream: This issue tracks a feature for terraphim/terraphim-ai

## Problem Statement Session search currently uses naive in-memory string matching (`String::contains`) which is O(n) over all messages and does not support relevance ranking, phrase queries, or fuzzy matching. The `terraphim_sessions/src/index/` directory exists but is empty. ## Current State - Search implementation: `crates/terraphim_sessions/src/service.rs:181` (linear scan) - Empty index module: `crates/terraphim_sessions/src/index/` (no files) - No Tantivy dependency in `terraphim_sessions/Cargo.toml` - Search returns all matches without ranking ## Acceptance Criteria - [ ] Add `tantivy` dependency to `terraphim_sessions/Cargo.toml` - [ ] Define Tantivy index schema with session/message fields - [ ] Implement `SessionIndex` with add/search methods - [ ] Integrate index into `SessionService` for automatic indexing on import - [ ] Search returns ranked results by relevance - [ ] Support phrase queries and prefix matching - [ ] Index persists to disk between sessions - [ ] cargo test -p terraphim_sessions passes - [ ] cargo clippy -- -D warnings passes ## Proposed Approach 1. Add `tantivy = "0.21"` to `terraphim_sessions/Cargo.toml` 2. Create `crates/terraphim_sessions/src/index/mod.rs` with: - `SessionIndexSchema` defining fields (session_id, message_id, content, code_content, timestamp, role, source) - `SessionIndex` struct with `add_session()` and `search()` methods - Custom tokenizers for code (edge n-gram) and natural language 3. Modify `SessionService` to maintain a `SessionIndex` alongside the cache 4. Update `search()` to use Tantivy when index is available, fallback to linear scan ## Dependencies None ## Verification ```bash cargo test -p terraphim_sessions cargo build -p terraphim_agent --features repl-sessions cargo clippy -p terraphim_sessions -- -D warnings ``` ## Scope New module (~300 lines) + modifications to `service.rs`. Single PR. Upstream: This issue tracks a feature for terraphim/terraphim-ai

root referenced this issue

2026-04-24 03:04:58 +02:00

feat: /sessions recommend subcommand for cross-session learning #9