Request: Embedded Prime RecallEngine documentation + integration guide

[decebal]

Context

We're building Longhand, a Tauri 2 desktop app that uses allsource-core (currently 0.13.1) with embedded + embedded-projections features for all persistence. We want to upgrade to 0.17.1 and adopt Prime for agent memory.

Current Pain

Our cross-project context retrieval does O(n) brute-force cosine similarity over all workflow outputs on every skill execution:

• Generates embedding via external OpenAI API call
• Loads all outputs from an AllSource projection
• Computes cosine similarity against every one
• Returns top 5

This works at small scale but degrades at thousands of outputs, and depends on an external embedding API.

What We Want From Prime

1. Embedded RecallEngine — use prime-full or prime-recall feature to get tiered retrieval (L0/L1/L2) without running a separate MCP server
2. Replace O(n) cosine search with HNSW via VectorIndexProjection
3. Local embeddings via fastembed to eliminate external API dependency
4. Cross-execution learning via graph projections

Requests

1. Documentation for embedded Prime usage — the current docs focus on the MCP server (allsource-prime crate). We need a guide for using Prime::open() + RecallEngine as an embedded library within a Tauri app.

2. Migration guide from EmbeddedCore to Prime — can Prime::from_core() wrap an existing EmbeddedCore instance? Or do we need to initialize Prime separately and register its projections alongside our existing 39 domain projections?

3. VectorIndexProjection usage example — how to ingest embeddings, query by similarity, and integrate with the existing projection system.

4. fastembed model selection — which model is recommended for short-form content (skill outputs, 100-500 chars)? What's the cache size impact?

5. Tiered recall configuration — how to configure L0/L1/L2 token budgets and what IndexConfig options are available?

Environment

• allsource-core 0.13.1 → upgrading to 0.17.1
• allframe-core 0.1.25 / allframe-tauri 0.1.25
• Rust 1.94.0, edition 2021
• 39 domain-specific projections registered on EmbeddedCore
• macOS desktop app (Tauri 2)

Relevant Code

Our current brute-force search (~70 lines we want to replace with a single RecallEngine call):

pub(crate) async fn retrieve_cross_project_context(
    skill_prompt: &str,
    project_scope: Option<&[String]>,
) -> Option<String> {
    let all_outputs = workflow_outputs_store::list_all_outputs();
    if all_outputs.is_empty() { return None; }
    
    let embed_config = embedding::EmbeddingConfig::from_env_async().await;
    let query_embedding = embedding::generate_embedding(skill_prompt, &embed_config).await?;
    
    // O(n) cosine similarity over ALL outputs
    let mut candidates: Vec<(f64, String)> = Vec::new();
    for output in all_outputs {
        if output.embedding.is_empty() { continue; }
        let similarity = cosine_similarity(&query_embedding, &output.embedding);
        candidates.push((similarity, output.content));
    }
    candidates.sort_by(|a, b| b.0.partial_cmp(&a.0).unwrap());
    candidates.truncate(5);
    // ... format as context string
}

We'd love to replace this with:

let context = recall.context(skill_prompt, ContextTier::L2).await?;

[decebal]

Embedded Prime Integration Guide

Created docs/guides/EMBEDDED_PRIME_INTEGRATION.md addressing all five requests:

1. Documentation for embedded Prime usage

• Prime::open(path) for durable init, Prime::open_in_memory() for testing
• Tauri-specific setup example with app.manage(Arc::new(prime))
• Feature flags table: prime-full + prime-recall is what you want

2. Migration guide from EmbeddedCore to Prime

• Prime wraps EmbeddedCore — prime.core() gives access to the underlying store
• Three migration options documented:
  • Option A: Prime owns the Core (new projects)
  • Option B: Register your 39 domain projections on Prime's underlying store via prime.core().inner() + register_projection_with_backfill()
  • Option C: Separate data directories for full isolation

3. VectorIndexProjection usage example

• prime.embed(id, text, embedding) to store
• prime.vector_search(&query_vec, top_k) for O(log n) HNSW search
• prime.vector_search_cross_domain() for round-robin interleaving across domains
• prime.similar(id, top_k) for find-similar

4. fastembed model selection

• Recommendation: AllMiniLmL6V2 (default) — 384 dims, ~23 MB, optimized for short passages
• Direct fastembed usage example + VectorSearchEngine wrapper
• Model comparison table for short-form content

5. Tiered recall configuration

• L0 (~100-200 tokens): Stats only, O(1)
• L1 (~500-1500 tokens): Conversation-scoped nodes + 1-hop edges
• L2 (~2000-5000 tokens): Full hybrid — compressed index + vectors + graph expansion
• IndexConfig reference with all fields documented
• Optional Ollama LLM compression vs deterministic heuristic

The 70-line replacement

// Before: O(n) brute-force cosine similarity + external API
let embed_config = embedding::EmbeddingConfig::from_env_async().await;
let query_embedding = embedding::generate_embedding(skill_prompt, &embed_config).await?;
// ... 70 lines of manual cosine similarity

// After: O(log n) HNSW + local embeddings + tiered recall
let ctx = recall.context(RecallContextQuery {
    query: skill_prompt.to_string(),
    tier: ContextTier::L2,
    top_k: 5,
    ..Default::default()
}).await;

Complete Tauri integration example included at the end of the guide.