Data-First: RAG vs Tools boundary¶

Principle: Never let an LLM invent engineering numbers. Route data by type before the agent answers.

Canonical maturity context: roadmap Stage 1–2. Design rules: principles.

Why this matters¶

RAG works well on narrative text (manual procedures, API descriptions, design rationale). It fails on tabular numeric facts — yield strength, elastic modulus, section catalogs — because chunking breaks table structure and the model fills gaps with plausible but wrong values.

Data-First means: classify the data type first, then pick the retrieval path.

Routing table¶

Data type	Path	Examples	Why
Narrative / requirements	RAG + mandatory citation	Manual sections, tutorials, API docs	Meaning + auditable source
Tabular / numeric facts	Structured Tools or API	σ_y, E, section properties, material catalogs	No hallucinated digits
Derived engineering results	Deterministic tools	Particle count, unit conversion, timestep heuristic	Unit-tested arithmetic

Target decision flow (Stage 1 extension)¶

Status: Planned — route by data type (narrative vs tabular vs derived). Structured material lookup is not implemented yet; see Implemented runtime below.

flowchart TD
    Q[User question] --> Classify{Needs a number?}
    Classify -->|No — meaning or requirement| RAG[RAG node]
    Classify -->|Yes — material property or catalog lookup| ToolLookup[Structured tool / API]
    Classify -->|Yes — engineering calculation| CalcTool[Deterministic calc tool]

    RAG --> Cite[Citation in report]
    ToolLookup --> Validate[Pydantic validation]
    CalcTool --> Validate
    Validate --> Report[Report node]
    Cite --> Report

Implemented runtime (Stage 2)¶

Status: Shipped — route by intent (knowledge / setup / scripting / calculation / unknown). One request takes a single branch; there is no combined RAG + tools path yet.

flowchart TD
    Q[User query] --> I[Intent node]
    I -->|knowledge setup scripting| RAG[RAG node — retrieval only]
    I -->|calculation| Tools[Tools node — deterministic calcs]
    I -->|unknown| Report[Report node]
    RAG --> Report
    Tools --> Report

Rocky manual content is retrieved via RAG with citations, not a structured catalog API.

Implemented today (Stage 2)¶

Path	Code	Status
RAG + citations	`nodes/rag.py`, `rag/chunking.py`, `rag/retriever.py`	Done
Vector + metadata filters	Cosine search + Qdrant payload filters (`doc_type`, manifest metadata)	Done
Deterministic calcs	`tools/*.py` — DEM helpers (particle count, units, timestep)	Done
Validation retry	Pydantic schemas + up to 3 tools-node retries (LLM re-invoked)	Done
Citation guardrail	`CAEC_REQUIRE_CITATIONS`	Done

Planned (Stage 1 extension): BM25 + vector fusion; move tabular catalogs to structured stores and tool lookups after Unstructured.io ingestion normalizes PDF tables.

Stage 1 — shipped vs planned¶

Component	Role	Status
PyMuPDF + manifest	Rocky v26r1 PDF ingest, page-aware chunks	Done (MVP)
Qdrant	Vector index with parent/child chunks	Done
Metadata contracts	`doc_type`, `cae_product`, version, source doc ID	Done (manifest)
Unstructured.io	PDF/GOST parsing, table extraction	Planned
Structured material API	Tool lookup for σ_y, E, catalog rows	Planned
BM25 + vector fusion	Full hybrid retrieval	Planned

Anti-patterns¶

Anti-pattern	Risk	Correct approach
Ask RAG for a yield strength from a table	Hallucinated MPa value	Structured tool or API lookup
Let LLM estimate particle count	Arithmetic errors	`estimate_particle_count()` pure function
Skip citation on manual requirement	Compliance / rework risk	Mandatory citation in report node
Ingest chaos, then add agents	"AI on top of chaos"	Stage 1 data foundation first

Data flow — RAG sequence (implemented)
Runtime view — tool-calling and validation loop
Principles — pure tools, citations, vector + metadata retrieval
Roadmap — Stage 1 Data Foundation