Biblicus

Make your documents usable by your assistant, then decide later how you will search and retrieve them.

If you are building an assistant in Python, you probably have material you want it to use: notes, documents, web pages, and reference files. A common approach is retrieval augmented generation, where a system retrieves relevant material and uses it as evidence when generating a response.

The first practical problem is not retrieval. It is collection and care. You need a stable place to put raw items, you need a small amount of metadata so you can find them again, and you need a way to evolve your retrieval approach over time without rewriting ingestion.

Biblicus gives you a normal folder on disk to manage. In Biblicus documentation, that managed folder is called a corpus (plural: corpora). It stores each ingested item as a file, with optional metadata stored next to it. You can open and inspect the raw files directly. Any derived catalog or index can be rebuilt from the raw files.

It can be used alongside LangGraph, Tactus, Pydantic AI, any agent framework, or your own setup. Use it from Python or from the command line interface.

See retrieval augmented generation overview for a short introduction to the idea.

Web Dashboard

The Dashboard app provides a real-time web interface for viewing and managing your corpus data, powered by AWS Amplify Gen 2. Features include:

• Browse corpus catalog items with filtering by tags and media type
• View extraction snapshots and track progress in real-time
• Automatic catalog sync after extraction runs
• GSAP-animated, responsive React interface

See apps/dashboard/README.md for setup and deployment instructions.

Analysis highlights

• biblicus analyze markov learns a directed, weighted state transition graph over segmented text.
• YAML configurations support cascading composition plus dotted --config key=value overrides.
• Text extract splits long texts with an LLM by inserting XML tags in-place for structured spans.
• See docs/markov-analysis.md for Markov analysis details and runnable demos.
• See docs/text-extract.md for the text extract utility and examples.
• Graph extraction supports deterministic NLP baselines (NER entities, dependency relations) for graph-aware retrieval experiments.
• Graph extractor demos are runnable from the repository (single extractor or all extractors).

Start with a knowledge base

If you just want to hand a folder to your assistant and move on, use the high-level knowledge base interface. The folder can be nothing more than a handful of plain text files. You are not choosing a retrieval strategy yet. You are just collecting.

This example assumes a folder called notes/ with a few .txt files. The knowledge base handles sensible defaults and still gives you a clear context pack for your model call.

from biblicus.knowledge_base import KnowledgeBase


kb = KnowledgeBase.from_folder("notes")
result = kb.query("Primary button style preference")
context_pack = kb.context_pack(result, max_tokens=800)

print(context_pack.text)

If you want to run a real, executable version of this story, use scripts/readme_end_to_end_demo.py from a fresh clone.

This simplified sequence diagram shows the same idea at a high level.

%%{init: {"theme": "base", "themeVariables": {"background": "#ffffff", "primaryColor": "#f3e5f5", "primaryTextColor": "#111111", "primaryBorderColor": "#8e24aa", "lineColor": "#90a4ae", "secondaryColor": "#eceff1", "tertiaryColor": "#ffffff", "noteBkgColor": "#ffffff", "noteTextColor": "#111111", "actorBkg": "#f3e5f5", "actorBorder": "#8e24aa", "actorTextColor": "#111111"}}}%%
sequenceDiagram
  participant App as Your assistant code
  participant KB as Knowledge base
  participant LLM as Large language model

  App->>KB: query
  KB-->>App: evidence and context
  App->>LLM: context plus prompt
  LLM-->>App: response draft

Loading

A simple mental model

Think in three stages.

• Ingest puts raw items into a corpus. This is file first and human inspectable.
• Extract turns items into usable text. This is where you would do text extraction from Portable Document Format files, optical character recognition for images, or speech to text for audio. If an item is already text, extraction can simply read it. Extraction outputs are derived artifacts, not edits to the raw files.
• Retrieve searches extracted text and returns evidence. Evidence is structured so you can turn it into context for your model call in whatever way your project prefers.

Workflow planning (load → extract → index)

Biblicus exposes dependency planning in biblicus.workflow. It builds deterministic task plans that describe which steps must run before a query or index operation. You can use it to check whether a corpus needs loading, extraction, or indexing and execute those steps in order.

from biblicus.workflow import build_default_handler_registry, build_plan_for_index

plan = build_plan_for_index(corpus, retriever_id="tf-vector")
results = plan.execute(mode="auto", handler_registry=build_default_handler_registry(corpus))

If you learn a few project words, the rest of the system becomes predictable.

• Corpus is the managed folder that holds raw items and their metadata.
• Item is the raw bytes plus optional metadata and source information.
• Catalog is the rebuildable index of the corpus.
• Extraction run is a recorded extraction build that produces text artifacts.
• Backend is a pluggable retrieval implementation.
• Run is a recorded retrieval build for a corpus.
• Evidence is what retrieval returns, with identifiers and source information.

Where it fits in an assistant

Biblicus does not answer user questions. It is not a language model. It helps your assistant answer them by retrieving relevant material and returning it as structured evidence. Your code decides how to turn evidence into a context pack for the model call, which is then passed to a model you choose.

In a coding assistant, retrieval is often triggered by what the user is doing right now. For example: you are about to propose a user interface change, so you retrieve the user's stated preferences, then you include that as context for the model call.

This diagram shows two sequential Biblicus calls. They are shown separately to make the boundaries explicit: retrieval returns evidence, and context pack building consumes evidence.

%%{init: {"theme": "base", "themeVariables": {"background": "#ffffff", "primaryColor": "#f3e5f5", "primaryTextColor": "#111111", "primaryBorderColor": "#8e24aa", "lineColor": "#90a4ae", "secondaryColor": "#eceff1", "tertiaryColor": "#ffffff", "noteBkgColor": "#ffffff", "noteTextColor": "#111111", "actorBkg": "#f3e5f5", "actorBorder": "#8e24aa", "actorTextColor": "#111111"}}}%%
sequenceDiagram
  participant User
  participant App as Your assistant code
  participant Bib as Biblicus
  participant LLM as Large language model

  User->>App: request
  App->>Bib: query retrieval
  Bib-->>App: retrieval result evidence JSON
  App->>Bib: build context pack from evidence
  Bib-->>App: context pack text
  App->>LLM: context pack plus prompt
  LLM-->>App: response draft
  App-->>User: response

Loading

Practical value

• You can ingest raw material once, then try many retrieval approaches over time.
• You can keep raw files readable and portable, without locking your data inside a database.
• You can evaluate retrieval snapshots against shared datasets and compare backends using the same corpus.

Typical flow

• Initialize a corpus folder.
• Ingest items from file paths, web addresses, or text input.
• Crawl a website section into corpus items when you want a repeatable “import from the web” workflow.
• Run extraction when you want derived text artifacts from non-text sources.
• Reindex to refresh the catalog after edits.
• Build a retrieval snapshot with a backend.
• Query the run to collect evidence and evaluate it with datasets.

Install

This repository is a working Python package. Install it into a virtual environment from the repository root.

python -m pip install -e .

After the first release, you can install it from Python Package Index.

python -m pip install biblicus

Optional extras

Some extractors are optional so the base install stays small.

• Optical character recognition for images: python -m pip install "biblicus[ocr]"
• Advanced optical character recognition with PaddleOCR: python -m pip install "biblicus[paddleocr]"
• Document understanding with Docling VLM: python -m pip install "biblicus[docling]"
• Document understanding with Docling VLM and MLX acceleration: python -m pip install "biblicus[docling-mlx]"
• Speech to text transcription with OpenAI: python -m pip install "biblicus[openai]" (requires an OpenAI API key in ~/.biblicus/config.yml or ./.biblicus/config.yml)
• Speech to text transcription with Deepgram: python -m pip install "biblicus[deepgram]" (requires a Deepgram API key in ~/.biblicus/config.yml or ./.biblicus/config.yml)
• Speech to text transcription with Aldea: python -m pip install "biblicus[aldea]" (requires ALDEA_API_KEY or aldea.api_key in ~/.biblicus/config.yml or ./.biblicus/config.yml)
• Broad document parsing fallback: python -m pip install "biblicus[unstructured]"
• MarkItDown document conversion (requires Python 3.10 or higher): python -m pip install "biblicus[markitdown]"
• Topic modeling analysis with BERTopic: python -m pip install "biblicus[topic-modeling]"

Quick start

mkdir -p notes
echo "A small file note" > notes/example.txt

biblicus init corpora/example
biblicus ingest --corpus corpora/example notes/example.txt
echo "A short note" | biblicus ingest --corpus corpora/example --stdin --title "First note"
biblicus list --corpus corpora/example
biblicus extract build --corpus corpora/example --stage pass-through-text --stage metadata-text
biblicus extract list --corpus corpora/example
biblicus build --corpus corpora/example --backend scan
biblicus query --corpus corpora/example --query "note"

Web Ingestion

Biblicus supports ingesting content directly from the web using two approaches.

Ingest from URLs

Ingest individual documents or web pages from URLs. The ingest command automatically detects content types including PDF, HTML, Markdown, images, and audio:

# Ingest a document from a URL
biblicus ingest https://example.com/document.pdf --tags "research"

# Ingest a web page
biblicus ingest https://example.com/article.html --tags "article"

# Ingest with a corpus path specified
biblicus ingest --corpus corpora/example https://docs.example.com/guide.md --tags "documentation"

Crawl Websites

Crawl entire website sections with automatic link discovery. The crawler follows links within the allowed prefix and stores discovered content:

# Crawl a documentation site
biblicus crawl \
  --corpus corpora/example \
  --root-url https://docs.example.com/ \
  --allowed-prefix https://docs.example.com/ \
  --max-items 100 \
  --tags "documentation"

# Crawl a specific blog category
biblicus crawl \
  --corpus corpora/example \
  --root-url https://blog.example.com/category/tutorials/ \
  --allowed-prefix https://blog.example.com/category/tutorials/ \
  --max-items 50 \
  --tags "tutorials,blog"

The --allowed-prefix parameter restricts the crawler to only follow links that start with the specified URL prefix, preventing it from crawling outside the intended scope. The crawler respects .biblicusignore rules and stores items under imports/crawl/ in your corpus.

End-to-end example: lower-level control

The command-line interface returns JavaScript Object Notation by default. This makes it easy to use Biblicus in scripts and to treat retrieval as a deterministic, testable step.

This version shows the lower-level pieces explicitly. You are building the corpus, controlling each memory string, choosing the backend, and shaping the context pack yourself.

from biblicus.backends import get_backend
from biblicus.context import ContextPackPolicy, TokenBudget, build_context_pack, fit_context_pack_to_token_budget
from biblicus.corpus import Corpus
from biblicus.models import QueryBudget


corpus = Corpus.init("corpora/story")

notes = [
    ("User name", "The user's name is Tactus Maximus."),
    ("Button style preference", "Primary button style preference: the user's favorite color is magenta."),
    ("Style preference", "The user prefers concise answers."),
    ("Language preference", "The user dislikes idioms and abbreviations."),
    ("Engineering preference", "The user likes code that is over-documented and behavior-driven."),
]
for note_title, note_text in notes:
    corpus.ingest_note(note_text, title=note_title, tags=["memory"])

backend = get_backend("scan")
run = backend.build_run(corpus, configuration_name="Story demo", config={})
budget = QueryBudget(max_total_items=5, maximum_total_characters=2000, max_items_per_source=None)
result = backend.query(
    corpus,
    run=run,
    query_text="Primary button style preference",
    budget=budget,
)

policy = ContextPackPolicy(join_with="\n\n")
context_pack = build_context_pack(result, policy=policy)
context_pack = fit_context_pack_to_token_budget(
    context_pack,
    policy=policy,
    token_budget=TokenBudget(max_tokens=60),
)
print(context_pack.text)

If you want a runnable version of this story, use the script at scripts/readme_end_to_end_demo.py.

If you prefer the command-line interface, here is the same flow in compressed form:

biblicus init corpora/story
biblicus ingest --corpus corpora/story --stdin --title "User name" --tag memory <<< "The user's name is Tactus Maximus."
biblicus ingest --corpus corpora/story --stdin --title "Button style preference" --tag memory <<< "Primary button style preference: the user's favorite color is magenta."
biblicus ingest --corpus corpora/story --stdin --title "Style preference" --tag memory <<< "The user prefers concise answers."
biblicus ingest --corpus corpora/story --stdin --title "Language preference" --tag memory <<< "The user dislikes idioms and abbreviations."
biblicus ingest --corpus corpora/story --stdin --title "Engineering preference" --tag memory <<< "The user likes code that is over-documented and behavior-driven."
biblicus build --corpus corpora/story --backend scan
biblicus query --corpus corpora/story --query "Primary button style preference"

Example output:

{
  "query_text": "Primary button style preference",
  "budget": {
    "max_total_items": 5,
    "maximum_total_characters": 2000,
    "max_items_per_source": null
  },
  "snapshot_id": "RUN_ID",
  "configuration_id": "RECIPE_ID",
  "backend_id": "scan",
  "generated_at": "2026-01-29T00:00:00.000000Z",
  "evidence": [
    {
      "item_id": "ITEM_ID",
      "source_uri": "text",
      "media_type": "text/markdown",
      "score": 1.0,
      "rank": 1,
      "text": "Primary button style preference: the user's favorite color is magenta.",
      "content_ref": null,
      "span_start": null,
      "span_end": null,
      "stage": "scan",
      "configuration_id": "RECIPE_ID",
      "snapshot_id": "RUN_ID",
      "hash": null
    }
  ],
  "stats": {}
}

Evidence is the output contract. Your code decides how to convert evidence into assistant context.

Turn evidence into a context pack

A context pack is a readable text block you send to a model. There is no single correct format. Treat it as a policy surface you can iterate on.

Here is a minimal example that builds a context pack from evidence:

from biblicus.context import ContextPackPolicy, build_context_pack


policy = ContextPackPolicy(
    join_with="\n\n",
)
context_pack = build_context_pack(result, policy=policy)
print(context_pack.text)

Example context pack output:

Primary button style preference: the user's favorite color is magenta.

You can also build a context pack from the command-line interface by piping the retrieval result:

biblicus query --corpus corpora/story --query "Primary button style preference" \\
  | biblicus context-pack build

Most production systems also apply a budget when building context. If you want a precise token budget, the budgeting logic needs a specific tokenizer and should be treated as its own stage.

Pipeline diagram

This diagram shows how a corpus becomes evidence for your assistant. Your code decides how to turn evidence into context and how to call a model.

%%{init: {"theme": "base", "themeVariables": {"primaryColor": "#f3e5f5", "primaryTextColor": "#111111", "primaryBorderColor": "#8e24aa", "lineColor": "#90a4ae", "secondaryColor": "#eceff1", "tertiaryColor": "#ffffff"}, "flowchart": {"useMaxWidth": true, "nodeSpacing": 18, "rankSpacing": 22}}}%%
flowchart TB
  subgraph Legend[Legend]
    direction LR
    LegendArtifact[Stored artifact or evidence]
    LegendStep[Step]
    LegendArtifact --- LegendStep
  end

  subgraph Main[" "]
    direction TB

    subgraph Pipeline[" "]
      direction TB

      subgraph RowStable[Stable core]
        direction TB
        Source[Source items] --> Ingest[Ingest] --> Raw[Raw item files] --> Catalog[Catalog file]
      end

      subgraph RowExtraction[Pluggable: extraction pipeline]
        direction TB
        Catalog --> Extract[Extract pipeline] --> ExtractedText[Extracted text artifacts] --> ExtractionRun[Extraction snapshot manifest]
      end

      subgraph RowRetrieval[Pluggable: retrieval backend]
        direction TB
        ExtractionRun --> Build[Build run] --> BackendIndex[Backend index] --> Run[Run manifest] --> Retrieve[Retrieve] --> Rerank[Rerank optional] --> Filter[Filter optional] --> Evidence[Evidence]
      end

      subgraph RowContext[Context]
        direction TB
        Evidence --> ContextPack[Context pack] --> FitTokens[Fit tokens optional] --> Context[Assistant context]
      end

      subgraph RowYourCode[Your code]
        direction TB
        Context --> Model[Large language model call] --> Answer[Answer]
      end
    end

    style RowStable fill:#ffffff,stroke:#8e24aa,stroke-width:2px,color:#111111
    style RowExtraction fill:#ffffff,stroke:#5e35b1,stroke-dasharray:6 3,stroke-width:2px,color:#111111
    style RowRetrieval fill:#ffffff,stroke:#1e88e5,stroke-dasharray:6 3,stroke-width:2px,color:#111111
    style RowContext fill:#ffffff,stroke:#7b1fa2,stroke-width:2px,color:#111111
    style RowYourCode fill:#ffffff,stroke:#d81b60,stroke-width:2px,color:#111111

    style Raw fill:#f3e5f5,stroke:#8e24aa,color:#111111
    style Catalog fill:#f3e5f5,stroke:#8e24aa,color:#111111
    style ExtractedText fill:#f3e5f5,stroke:#8e24aa,color:#111111
    style ExtractionRun fill:#f3e5f5,stroke:#8e24aa,color:#111111
    style BackendIndex fill:#f3e5f5,stroke:#8e24aa,color:#111111
    style Run fill:#f3e5f5,stroke:#8e24aa,color:#111111
    style Evidence fill:#f3e5f5,stroke:#8e24aa,color:#111111
    style ContextPack fill:#f3e5f5,stroke:#8e24aa,color:#111111
    style Context fill:#f3e5f5,stroke:#8e24aa,color:#111111
    style Answer fill:#f3e5f5,stroke:#8e24aa,color:#111111
    style Source fill:#f3e5f5,stroke:#8e24aa,color:#111111

    style Ingest fill:#eceff1,stroke:#90a4ae,color:#111111
    style Extract fill:#eceff1,stroke:#90a4ae,color:#111111
    style Build fill:#eceff1,stroke:#90a4ae,color:#111111
    style Retrieve fill:#eceff1,stroke:#90a4ae,color:#111111
    style Rerank fill:#eceff1,stroke:#90a4ae,color:#111111
    style Filter fill:#eceff1,stroke:#90a4ae,color:#111111
    style FitTokens fill:#eceff1,stroke:#90a4ae,color:#111111
    style Model fill:#eceff1,stroke:#90a4ae,color:#111111
  end

  style Legend fill:#ffffff,stroke:#ffffff,color:#111111
  style Main fill:#ffffff,stroke:#ffffff,color:#111111
  style Pipeline fill:#ffffff,stroke:#ffffff,color:#111111
  style LegendArtifact fill:#f3e5f5,stroke:#8e24aa,color:#111111
  style LegendStep fill:#eceff1,stroke:#90a4ae,color:#111111

Loading

Python usage

From Python, the same flow is available through the Corpus class and backend interfaces. The public surface area is small on purpose.

• Create a corpus with Corpus.init or open one with Corpus.open.
• Ingest notes with Corpus.ingest_note.
• Ingest files or web addresses with Corpus.ingest_source.
• List items with Corpus.list_items.
• Build a retrieval snapshot with get_backend and backend.build_run.
• Query a run with backend.query.
• Evaluate with evaluate_run.

Benchmarking

Biblicus is designed as a retrieval augmented generation platform where you can experiment with different extraction pipelines and benchmark them against each other.

Quick Start

# Download benchmark dataset
python scripts/download_funsd_samples.py

# Run benchmark on all pipelines
python scripts/benchmark_all_pipelines.py

# View results
cat results/final_benchmark.json | jq '.pipelines[] | {name, f1: .metrics.set_based.avg_f1}'

Available Benchmarks

• Forms (FUNSD): 199 scanned form documents with ground truth
• Receipts (SROIE): 626 receipt images for entity extraction
• Academic Papers (Scanned ArXiv): Multi-column layout understanding (dataset pending)

Evaluation Metrics

• Set-based: Precision, Recall, F1 Score (position-agnostic accuracy)
• Order-aware: Word Error Rate, LCS Ratio (reading order quality)
• N-gram: Bigram and trigram overlap (local ordering)

Pipeline Comparison

Pipeline	F1 Score	Recall	Use Case
PaddleOCR	0.787	0.782	Best overall accuracy
Docling-Smol	0.728	0.675	Tables & formulas
Heron + Tesseract	0.519	0.810	Maximum extraction

See the Benchmarking Overview for complete documentation.

Learn more

Full documentation is published on GitHub Pages: https://anthusai.github.io/Biblicus/

The documents below follow the pipeline from raw items to model context:

• Corpus
• Text extraction
• Speech to text
• Knowledge base
• Backends
• Context packs
• Testing and evaluation
• Benchmarking

Reference:

• Demos
• User configuration

Design and implementation map:

• Feature index
• Roadmap
• Architecture

Metadata and catalog

Raw items are stored as files directly under the corpus root. Metadata can live in a Markdown front matter block or a sidecar file with the suffix .biblicus.yml. The catalog lives in metadata/catalog.json and can be rebuilt at any time with biblicus reindex.

Corpus layout

corpus/
  metadata/
    config.json
    catalog.json
  extracted/
    pipeline/
      <snapshot id>/
        manifest.json
        text/
          <item id>.txt
  retrieval/
    <backend id>/
      <snapshot id>/
        manifest.json
  item.bin
  item.bin.biblicus.yml

Retrieval backends

Three backends are included.

• scan is a minimal baseline that scans raw items directly.
• sqlite-full-text-search is a practical baseline that builds a full text search index in SQLite.
• tf-vector is a deterministic term-frequency vector baseline with cosine similarity scoring.

For detailed documentation including configuration options, performance characteristics, and usage examples, see the Backend Reference.

Retrieval documentation

For the retrieval pipeline overview and snapshot artifacts, see docs/retrieval.md. For retrieval quality upgrades (tuned lexical baseline, reranking, hybrid retrieval), see docs/retrieval-quality.md. For evaluation workflows and dataset formats, see docs/retrieval-evaluation.md. For a runnable walkthrough, use the retrieval evaluation lab script (scripts/retrieval_evaluation_lab.py).

Extraction backends

These extractors are built in. Optional ones require extra dependencies. See text extraction documentation for details.

Text and document extraction

• pass-through-text reads text items and strips Markdown front matter.
• metadata-text turns catalog metadata into a small text artifact.
• pdf-text extracts text from Portable Document Format items with pypdf.
• unstructured provides broad document parsing (optional).
• markitdown converts many formats into Markdown-like text (optional).

Optical character recognition

• ocr-rapidocr does optical character recognition on images (optional).
• ocr-paddleocr-vl does advanced optical character recognition with PaddleOCR vision-language model (optional).

Vision-language models

• docling-smol uses the SmolDocling-256M vision-language model for fast document understanding (optional).
• docling-granite uses the Granite Docling-258M vision-language model for high-accuracy extraction (optional).

Speech to text

• stt-openai performs speech to text on audio using OpenAI (optional).
• stt-deepgram performs speech to text on audio using Deepgram (optional).
• stt-aldea performs speech to text on audio using Aldea (optional).

Pipeline utilities

• select-text chooses one prior extraction result in a pipeline.
• select-longest-text chooses the longest prior extraction result.
• select-override chooses the last extraction result for matching media types in a pipeline.
• select-smart-override intelligently chooses between extraction results based on confidence and content quality.

For detailed documentation on all extractors, see the Extractor Reference.

For extraction evaluation workflows, dataset formats, and report interpretation, see docs/extraction-evaluation.md.

Text extract utility

Text extract is a reusable analysis utility that lets a model insert XML tags into a long text without re-emitting the entire document. It returns structured spans and the marked-up text, and it is used as a segmentation option in Markov analysis.

See docs/text-extract.md for the utility API and examples, and docs/markov-analysis.md for the Markov integration.

Text slice utility

Text slice is a reusable analysis utility that lets a model insert <slice/> markers into a long text without re-emitting the entire document. It returns ordered slices and the marked-up text for auditing and reuse.

See docs/text-slice.md for the utility API and examples.

Topic modeling analysis

Biblicus can run analysis pipelines on extracted text without changing the raw corpus. Profiling and topic modeling are the first analysis backends. Profiling summarizes corpus composition and extraction coverage. Topic modeling reads an extraction snapshot, optionally applies an LLM-driven extraction pass, applies lexical processing, runs BERTopic, and optionally applies an LLM fine-tuning pass to label topics. The output is structured JavaScript Object Notation.

See docs/analysis.md for the analysis pipeline overview, docs/profiling.md for profiling, and docs/topic-modeling.md for topic modeling details.

Run a topic analysis using a configuration file:

biblicus analyze topics --corpus corpora/example --configuration configurations/topic-modeling.yml --extraction-run pipeline:<snapshot_id>

If --extraction-run is omitted, Biblicus uses the most recent extraction snapshot and emits a warning about reproducibility. The analysis output is stored under:

analysis/topic-modeling/<snapshot_id>/output.json

Minimal configuration example:

schema_version: 1
text_source:
  sample_size: 200
llm_extraction:
  enabled: false
lexical_processing:
  enabled: true
  lowercase: true
  strip_punctuation: false
  collapse_whitespace: true
bertopic_analysis:
  parameters:
    min_topic_size: 8
    nr_topics: 10
  vectorizer:
    ngram_range: [1, 2]
    stop_words: english
llm_fine_tuning:
  enabled: false

LLM extraction and fine-tuning require biblicus[openai] and a configured OpenAI API key. Configuration files are validated strictly against the topic modeling schema, so type mismatches or unknown fields are errors. AG News integration runs require biblicus[datasets] in addition to biblicus[topic-modeling].

For a repeatable, real-world integration run that downloads AG News and executes topic modeling, use:

python scripts/topic_modeling_integration.py --corpus corpora/ag_news_demo --force

See docs/topic-modeling.md for parameter examples and per-topic output behavior.

Integration corpus and evaluation dataset

Use scripts/download_ag_news.py to download the AG News dataset when running topic modeling demos. The repository does not include that content.

Use scripts/download_pdf_samples.py to download a small Portable Document Format integration corpus when running tests or demos. The repository does not include that content.

Tests and coverage

python scripts/test.py

To include integration scenarios that download public test data at runtime, run this command.

python scripts/test.py --integration

Releases

Releases are automated from the main branch using semantic versioning and conventional commit messages.

The release pipeline publishes a GitHub release and uploads the package to Python Package Index when continuous integration succeeds.

Publishing uses a Python Package Index token stored in the GitHub secret named PYPI_TOKEN.

Documentation

Reference documentation is generated from Sphinx style docstrings.

Install development dependencies:

python -m pip install -e ".[dev]"

Build the documentation:

python -m sphinx -b html docs docs/_build/html

Preview the documentation locally:

cd docs/_build/html
python -m http.server

Open http://localhost:8000 in your browser.

License

License terms are in LICENSE.