feat: wildcard/relaxed query fallback when primary search returns zero results (F1.1) #29

New Issue

2026-04-25T05:03:04+02:00

product-owner commented

2026-04-25 05:03:04 +02:00

Problem Statement

Spec F1.1 (terraphim-agent-session-search-spec.md, lines 482-509) defines the canonical robot-mode response with meta.wildcard_fallback: bool. Issue #21 wires the meta field, but the underlying behaviour -- "fall back to a wildcard / relaxed query when the primary search returns zero hits" -- is not implemented. Without it, the field can never be true, so the spec semantics are unobservable.

Verification:

rg "wildcard_fallback|fallback_search|relax_query" crates/terraphim_service/   -> 0 matches
rg "wildcard_fallback|fallback_search|relax_query" crates/terraphim_sessions/ -> 0 matches

The user-facing failure mode today: terraphim-agent --robot search "async DB pool exhaustion in tokio runtime" returns zero results when no document matches the full phrase, even though sessions exist for "async pool" and "tokio". Agents and humans alike must manually iterate, which violates G2 ("zero parse failures from typos") and degrades G1 (search latency on the perceived path; users effectively retry).

Acceptance Criteria

Add crates/terraphim_service/src/search/fallback.rs (or extend the search module already in service) implementing:

pub enum FallbackStrategy {
    None,
    WildcardSuffix,        // append `*` to each token (Tantivy phrase -> term wildcards)
    DropLowestScoringTerm, // remove the term with lowest IDF and retry
    ConceptExpansion,      // expand each token via terraphim_automata thesaurus
}
pub struct FallbackOutcome {
    pub strategy_used: Option<FallbackStrategy>,
    pub original_query: String,
    pub effective_query: String,
    pub results_added: usize,
}

Default order on zero-result primary search: WildcardSuffix -> DropLowestScoringTerm -> ConceptExpansion; stop at the first strategy that produces >= 1 hit
Each strategy is a pure function that takes the parsed query AST and returns a new query string; no I/O until the search engine re-runs
When fallback fires, meta.wildcard_fallback = Some(true) AND meta.fallback_strategy = Some("wildcard_suffix" | "drop_term" | "concept_expansion") is set on the response envelope
When primary search succeeds (>=1 hit), meta.wildcard_fallback = Some(false) and meta.fallback_strategy = None
CLI flag --no-fallback on search and /sessions search to disable; documented in robot/docs.rs so robot schemas search shows it
Wire into both:
- terraphim_service::search (document search path)
- terraphim_sessions::Sessionservice::search (session search path) -- depends on #5 Tantivy index landing first; until then, session-search fallback is a no-op stub returning FallbackOutcome::default()
Unit tests for each strategy in isolation: WildcardSuffix on "async pool" -> "async* pool*"; DropLowestScoringTerm picks the rarest token (use a fixture frequency table); ConceptExpansion uses a fixture thesaurus
Integration test: search a fixture corpus with a known zero-hit phrase, assert fallback fires and meta.fallback_strategy == "wildcard_suffix"
Integration test: search with --no-fallback, assert zero results and meta.wildcard_fallback == Some(false) even when fallbacks would have helped
cargo test -p terraphim_service search::fallback passes
cargo test -p terraphim_agent --features repl-full search_wildcard_fallback passes
cargo clippy -p terraphim_service -p terraphim_agent -- -D warnings passes

Proposed Approach

Strategies implemented as small pure functions; orchestrator in service.rs::search runs primary first, then iterates strategies on zero-hit
Concept expansion reuses terraphim_automata::find_matches against the active role thesaurus to expand each token to its top-3 synonyms; cap expanded query length to 32 tokens to avoid runaway
DropLowestScoringTerm uses term-frequency from the rolegraph (already loaded); fall back to first-token-removed if no rolegraph available
Add meta.fallback_strategy: Option<String> to ResponseMeta (separate from wildcard_fallback) so consumers can branch on which strategy ran -- see #21 for the meta-extension pattern
Pair with #21 (which adds concepts_matched + the boolean): this issue closes the loop by emitting the strategy name AND making wildcard_fallback = true actually achievable

Dependencies

Pairs with (does not block) #21 -- if #21 lands first, this issue extends ResponseMeta further; if this lands first, #21 just adds concepts_matched
Session-search fallback path depends on #5 (Tantivy index); landing this issue is fine while #5 is in flight -- session fallback is a no-op stub until then

Verification

cargo test -p terraphim_service search::fallback
cargo test -p terraphim_agent --features repl-full search_wildcard_fallback
cargo clippy -p terraphim_service -p terraphim_agent -- -D warnings
./target/debug/terraphim-agent --robot search "totally nonexistent phrase about quantum sockets" | jq '.meta.wildcard_fallback, .meta.fallback_strategy'
./target/debug/terraphim-agent --robot search "totally nonexistent phrase about quantum sockets" --no-fallback | jq '.meta.wildcard_fallback'

Expected first call: meta.wildcard_fallback == true AND fallback_strategy is one of wildcard_suffix|drop_term|concept_expansion. Second call: false, no strategy.

Scope

Single PR, ~400-500 LOC across terraphim_service::search::fallback (~200), service orchestrator wiring (~60), session-service stub (~20), CLI/docs (~30), tests (~150). Closes the F1.1 spec semantic gap left by #21.

Upstream: terraphim-agent-session-search-spec.md F1.1 (lines 482-509) -- complements #21.

## Problem Statement Spec F1.1 (terraphim-agent-session-search-spec.md, lines 482-509) defines the canonical robot-mode response with `meta.wildcard_fallback: bool`. Issue #21 wires the meta field, but the underlying behaviour -- "fall back to a wildcard / relaxed query when the primary search returns zero hits" -- is not implemented. Without it, the field can never be `true`, so the spec semantics are unobservable. Verification: ``` rg "wildcard_fallback|fallback_search|relax_query" crates/terraphim_service/ -> 0 matches rg "wildcard_fallback|fallback_search|relax_query" crates/terraphim_sessions/ -> 0 matches ``` The user-facing failure mode today: `terraphim-agent --robot search "async DB pool exhaustion in tokio runtime"` returns zero results when no document matches the full phrase, even though sessions exist for "async pool" and "tokio". Agents and humans alike must manually iterate, which violates G2 ("zero parse failures from typos") and degrades G1 (search latency on the perceived path; users effectively retry). ## Acceptance Criteria - [ ] Add `crates/terraphim_service/src/search/fallback.rs` (or extend the search module already in service) implementing: ```rust pub enum FallbackStrategy { None, WildcardSuffix, // append `*` to each token (Tantivy phrase -> term wildcards) DropLowestScoringTerm, // remove the term with lowest IDF and retry ConceptExpansion, // expand each token via terraphim_automata thesaurus } pub struct FallbackOutcome { pub strategy_used: Option<FallbackStrategy>, pub original_query: String, pub effective_query: String, pub results_added: usize, } ``` - [ ] Default order on zero-result primary search: `WildcardSuffix` -> `DropLowestScoringTerm` -> `ConceptExpansion`; stop at the first strategy that produces >= 1 hit - [ ] Each strategy is a pure function that takes the parsed query AST and returns a new query string; no I/O until the search engine re-runs - [ ] When fallback fires, `meta.wildcard_fallback = Some(true)` AND `meta.fallback_strategy = Some("wildcard_suffix" | "drop_term" | "concept_expansion")` is set on the response envelope - [ ] When primary search succeeds (>=1 hit), `meta.wildcard_fallback = Some(false)` and `meta.fallback_strategy = None` - [ ] CLI flag `--no-fallback` on `search` and `/sessions search` to disable; documented in `robot/docs.rs` so `robot schemas search` shows it - [ ] Wire into both: - `terraphim_service::search` (document search path) - `terraphim_sessions::Sessionservice::search` (session search path) -- depends on #5 Tantivy index landing first; until then, session-search fallback is a no-op stub returning `FallbackOutcome::default()` - [ ] Unit tests for each strategy in isolation: WildcardSuffix on `"async pool"` -> `"async* pool*"`; DropLowestScoringTerm picks the rarest token (use a fixture frequency table); ConceptExpansion uses a fixture thesaurus - [ ] Integration test: search a fixture corpus with a known zero-hit phrase, assert fallback fires and `meta.fallback_strategy == "wildcard_suffix"` - [ ] Integration test: search with `--no-fallback`, assert zero results and `meta.wildcard_fallback == Some(false)` even when fallbacks would have helped - [ ] `cargo test -p terraphim_service search::fallback` passes - [ ] `cargo test -p terraphim_agent --features repl-full search_wildcard_fallback` passes - [ ] `cargo clippy -p terraphim_service -p terraphim_agent -- -D warnings` passes ## Proposed Approach - Strategies implemented as small pure functions; orchestrator in `service.rs::search` runs primary first, then iterates strategies on zero-hit - Concept expansion reuses `terraphim_automata::find_matches` against the active role thesaurus to expand each token to its top-3 synonyms; cap expanded query length to 32 tokens to avoid runaway - DropLowestScoringTerm uses term-frequency from the rolegraph (already loaded); fall back to first-token-removed if no rolegraph available - Add `meta.fallback_strategy: Option<String>` to `ResponseMeta` (separate from `wildcard_fallback`) so consumers can branch on which strategy ran -- see #21 for the meta-extension pattern - Pair with #21 (which adds `concepts_matched` + the boolean): this issue closes the loop by emitting the strategy name AND making `wildcard_fallback = true` actually achievable ## Dependencies - Pairs with (does not block) #21 -- if #21 lands first, this issue extends `ResponseMeta` further; if this lands first, #21 just adds `concepts_matched` - Session-search fallback path depends on #5 (Tantivy index); landing this issue is fine while #5 is in flight -- session fallback is a no-op stub until then ## Verification ``` cargo test -p terraphim_service search::fallback cargo test -p terraphim_agent --features repl-full search_wildcard_fallback cargo clippy -p terraphim_service -p terraphim_agent -- -D warnings ./target/debug/terraphim-agent --robot search "totally nonexistent phrase about quantum sockets" | jq '.meta.wildcard_fallback, .meta.fallback_strategy' ./target/debug/terraphim-agent --robot search "totally nonexistent phrase about quantum sockets" --no-fallback | jq '.meta.wildcard_fallback' ``` Expected first call: `meta.wildcard_fallback == true` AND `fallback_strategy` is one of `wildcard_suffix|drop_term|concept_expansion`. Second call: `false`, no strategy. ## Scope Single PR, ~400-500 LOC across `terraphim_service::search::fallback` (~200), service orchestrator wiring (~60), session-service stub (~20), CLI/docs (~30), tests (~150). Closes the F1.1 spec semantic gap left by #21. Upstream: terraphim-agent-session-search-spec.md F1.1 (lines 482-509) -- complements #21.

product-owner added a new dependency 2026-04-25 05:03:12 +02:00

#5 feat: Implement Tantivy-based session index for full-text search

root referenced this issue

2026-04-25 05:07:16 +02:00

feat: redact api keys/secrets/JWTs during session import (Privacy) #26

product-owner referenced this issue

2026-04-25 11:09:06 +02:00

feat: criterion benchmark suite for G1 performance targets #43

product-owner referenced this issue

2026-04-27 07:00:23 +02:00

feat: wire ExitCode into main() return for f1.2 robot exit code contract #35

root referenced this issue