TTYG-178 Split documentation from README into pages by pgan002 · Pull Request #51 · Ontotext-AD/graphrag-eval

pgan002 · 2026-02-07T07:50:40Z

Move most of the documentation from the README file into separate files in a new directory docs/. Benefits:
1. The main page (README) is shorter, and so more welcoming
2. The main page loads faster
3. The sections are shorter and so easier to read
4. The directory helps to understand the contents
Major additions for completeness
Major edits clarity
Links to documentation sections from README and from other sections
Consistent section heading case: sentence case
Break text at 80 characters when possible

Tests

Spelling, grammar, typos: copy-pasted text into word processor
Links: manually followed each link in README.md and docs/*

TODO/questions:

Maybe re-join paragraphs into single long lines instead of breaking them
Usually, links to sections are titled "Section <full_section_name>" but in a few cases, the link title is a part of the sentence. Should we fix those?
Move section "Aggregate metrics" to a separate file or into section Metrics?
Some sections link to section with title "Inputs" but the section heading is "Input"

README.md

docs/usage.md

docs/installation.md

docs/steps-score.md

docs/retrieval-evaluation-using-chunk-ids.md

docs/0-intro.md

nelly-hateva · 2026-03-30T11:50:20Z

README.md

+* [Metrics](metrics.md)
+* [Installation](installation.md)
+* [Usage](usage.md)
+* [Configuration](documentation.md)
+* [LLM use in evaluation](llm-use-in-evaluation.md)
+* [Custom evaluation (custom metrics)](custom-evaluation.md)
+* [Input](input.md)
+* [Output](output.md)


These links don't work. If we want them to be relative, they should start with docs/. For example, docs/metrics.md. However, I think in order for the links to work on PYPI page https://pypi.org/project/graphrag-eval/ we should make them absolute, like so https://github.com/Ontotext-AD/graphrag-eval/blob/main/docs/metrics.md

Also "Configuration" links to a non existing md file documentation.md, it must link to the configuration.md

Fixed to relative links.

Relative links are shorter to read and edit, and work in an editor and a downloaded folder.

How useful is it for the documentation to be accessible from the PyPi page?

nelly-hateva · 2026-03-30T12:12:41Z

docs/installation.md

+```
+
+To evaluate `answer_relevance` and answer correctness metrics (`answer_recall`,
+`answer_precision`, `answer_f1`; see section [Output keys](output.md)) or


Suggested change

`answer_precision`, `answer_f1`; see section [Output keys](output.md)) or

`answer_precision`, `answer_f1) see section [Output keys](output.md) or

This suggestion fixes the formatting, but I also suggest to link to the metrics file instead of the output and define how the metrics are calculated in the metrics file

Why the suggestion? We have ( ... ; ... ).

nelly-hateva · 2026-03-30T12:20:10Z

docs/usage.md

+```
+
+`evaluation_results` is a list of statistics for each question, as in section
+[Example wvaluation results](examples/output.md). The format is explained in section


Suggested change

[Example wvaluation results](examples/output.md). The format is explained in section

[Example evaluation results](examples/output.md). The format is explained in section

Fixed, thanks. I should re-read, run a spell checker and maybe a LLM.

nelly-hateva · 2026-03-30T12:20:42Z

docs/usage.md

+[Example wvaluation results](examples/output.md). The format is explained in section
+[Output Keys](output.md)
+
+# Command Line Use


Suggested change

# Command Line Use

## Command Line Use

Why? Should we define top-level heading "Usage" and level-2 headings "Use as a library" and "Command-line use"?

nelly-hateva · 2026-03-30T12:24:10Z

docs/usage.md

+         reference dataset and `actual_answer` in the target data to evaluate
+      1. For custom metrics:
+         1. Define the metrics in the [configuration file](configuration.md)
+         1. Include refernce and target inputs used by the metrics


Suggested change

1. Include refernce and target inputs used by the metrics

1. Include reference and target inputs used by the metrics

Fixed. Thanks!

nelly-hateva · 2026-03-30T12:26:11Z

docs/usage.md

+      1. For `answer_relevance`, include `actual_answer` in the reference
+         dataset
+      1. For answer correctness metrics (section
+         [Output](output.md)), include `reference_answer` in the
+         reference dataset and `actual_answer` in the target data to evaluate
+      1. For custom metrics:


This is inconsistent, for the answer relevance and for the custom metrics we don't link to the definition, but for answer correctness we do. Also, I would define the metrics (except for the custom metrics) in the metrics file, and link to it.

Added links and made other changes to this section. Linked to metrics here.

As for changes to the contents of metrics.md in comments to that file.

Replaced the keyword by words without formatting, because a keyword suggests that the value is a key in the output.

pgan002 · 2026-03-30T23:33:42Z

@nelly-hateva please close threads.

pgan002 requested review from atagarev and nelly-hateva February 7, 2026 07:50

pgan002 force-pushed the TTYG-178 branch from 1fb54c2 to ba141bc Compare February 20, 2026 01:42

pgan002 closed this Mar 14, 2026

pgan002 force-pushed the TTYG-178 branch from ba141bc to e338970 Compare March 14, 2026 01:49

pgan002 reopened this Mar 14, 2026

pgan002 requested review from atagarev and nelly-hateva and removed request for atagarev and nelly-hateva March 14, 2026 02:51

pgan002 self-assigned this Mar 14, 2026