Add AGENTS.md

[Scienfitz]

AGENTS.md files contain content intended for agentic operators. They are recognized by most coding frameworks (most importantly claude and opencode) and are injected into the context whenever an agent reads a folder where such a file is contained. They lead to more consistent code being generated and generally more in line with what has already been done without explicitly having to state this over and over again.

The content here is meant as a start and not as complete. We can continue to add rules as we evolve.

The content has been produced in the following manner:

• Analyze the entire code base and the last 200 PRs including comments
• Extract from this rules and conventions that apply to this repo
• The resulting files have then been compressed
• I then did one coarse round of human review and curation

[Copilot]

Pull request overview

Adds AGENTS.md guidance files intended to be injected into agentic tooling context to steer code, tests, docs, and examples toward existing BayBE conventions.

Changes:

• Introduce root-level AI-agent coding guide (AGENTS.md) covering architecture, typing, imports, CI, and workflow.
• Add test-suite conventions for pytest structure/fixtures/parametrization (tests/AGENTS.md).
• Add docs and examples conventions for Sphinx/MyST and runnable scripts (docs/AGENTS.md, examples/AGENTS.md).

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 10 comments.

File	Description
AGENTS.md	Project-wide agent guidance for BayBE coding patterns, tooling, and PR workflow
tests/AGENTS.md	Conventions for writing/organizing pytest tests and fixtures
docs/AGENTS.md	Conventions for Sphinx/MyST docs authoring and syntax
examples/AGENTS.md	Conventions for executable examples and CI smoke-test behavior

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[m-aebrer]

I would add this rather in contributing.md instead of a tests specific agents.md, though I don't think that's a major issue.

I also usually add these test principles to get better results (they assume you have pre-commit hooks to run the tests, which you should):

---

Testing Conventions

Summary:

• Test through public interfaces: call functions, assert on return values and file system side effects
• Mock only external boundaries: the [INSERT BOUNDARY POINT HERE] is the boundary — provide a [SPECIFIC MOCKER HERE, WITH DEFINED NARROW SCOPE] that emits [PROJECT SPECIFIC DETAILS]
• One test, one behavior: don't combine assertions about different concerns
• Tests as specs: names like test_qc_fail_triggers_retry, test_missing_smiles_fails_validation

Run tests: python -m pytest backend/tests/

Pre-commit hook runs tests automatically before each commit. On a fresh clone, activate it once:

git config core.hooksPath .githooks

---

[Scienfitz]

total test suite runs 20-30 minutes so we cant run them in a high frequent manner such as pre-commit, would probably rather instruct to run tests relevant to the respective feature

[m-aebrer]

otherwise all looks good to me. I would say there may be some issues with lack of compliance for the anti-patterns as some of them are a bit underspecified, eg: No conftest pollution — prefer local fixtures. -> "pollution" would go better with examples of before and after for example

[m-aebrer]

ah @Scienfitz good point on too many tests. In that case I have CI run the full suite, and locally I have a subset using tags to only run those ones on pre-commit hook. This usually also prompts the agent naturally to run any task specific tests as well, as a reminder.

You can also just add a hook that only injects a warning/reminder about running tests into the chat context.

[m-aebrer]

seems like you have a no fallback rule.

I have run into issues with this before, here's my section about it:

## Zero Fallback Principle
Execution MUST abort immediately on any missing dependency, malformed data field, absent required column/key, unexpected enum value, or structural schema mismatch. Do **not** continue in a degraded or "best effort" mode. No silent defaults. No guessing. Expensive downstream computation must be prevented when prerequisites are not perfectly satisfied.


## Validation Philosophy
- **Per-Template Strict Validation**: Each template defines exact allowed factor choices; models cannot select factors outside their template.
- **Validation at Inference Time**: PromptSwarm validates outputs using the universal `ConditionRecommendationModel` with the template's `_allowed_choices` passed as validation context.
- **Zero Fallback**: Any validation failure TERMINATES that template's inference (circuit breaker pattern in promptswarm).
- **No Partial Results**: Invalid outputs are not written to the WFM database; no partial pipelines or continued processing.

Then I also have a hook that autodetects when Claude is trying to add a fallback and stops him:

https://github.com/merckgroup/condition_rec_benchmarking/blob/main/.claude/hooks/check-fallback.sh

[Scienfitz]

I've extracted some generalizable principles out of this and added to the file under the respective sections (590b521)

well discuss whether we can do something with the actual hook as well, in any case thanks for your input 🙌

[LeanAndMean]

Hey, nice work on the recent updates -- the fail-fast language in sections 2/8/16 reads well, and the admonition rewrite is much clearer.

I've been reviewing this alongside how I structure similar files in my repos and wanted to share a few thoughts. Happy to help implement any of these if useful, but also easy for you to pick up directly since they're mostly structural.

CLAUDE.md + symlinks: Have you considered making the canonical file CLAUDE.md? Claude Code automatically loads it into every conversation, and it supports subdirectory scoping -- a tests/CLAUDE.md is auto-injected when the agent works on files in tests/. That's a perfect match for the distributed structure you've set up here.

To keep other tools working, symlink from the root:

AGENTS.md -> CLAUDE.md
.github/copilot-instructions.md -> CLAUDE.md

Single source of truth, three consumers. The subdirectory files (tests/CLAUDE.md, etc.) don't need symlinks since the other tools don't have equivalent scoping.

Complementary content from /init: I ran Claude Code's /init command on this repo and it generated a CLAUDE.md that has some nice qualities that could complement what you've already built here:

• A full commands section covering install, test (pytest --fast, pytest -k "test_name"), lint, typecheck, and all tox environments -- including the python version append convention (tox -e mypy-py310) and tox -p for parallel runs. Your AGENTS.md covers conventions thoroughly but doesn't tell the agent how to validate its work; this fills that gap.
• The architecture section maps each domain concept to its file path (Campaign -> baybe/campaign.py, SearchSpace -> baybe/searchspace/, etc.), which gives an agent both "what is this" and "where to find it" in one pass.
• It distills design principles to the three an agent is most likely to violate: comp-rep boundary, lazy imports ("Non-negotiable"), and the serialization pattern. Useful as a complement to the fuller treatment in your AGENTS.md.
• It's 161 lines vs the current ~270, which matters for token budget since the root file is injected into every conversation.

I'm not suggesting replacing your work -- the AGENTS.md has valuable depth (validation patterns, deprecation strategy, the naming table, fixture architecture) that the /init output doesn't cover. But the commands and architecture-to-filepath mapping would be easy to fold in on your end. I can share the generated file if you'd like to compare.

Anti-pattern specificity: As m-aebrer mentioned, rules like "prefer local fixtures" are easier for agents to comply with when there's a concrete example. Even a one-liner showing the preferred pattern would help.

Let me know if any of this is useful -- happy to put together changes on a separate branch or contribute directly here, whichever you'd prefer.

[Scienfitz]

@LeanAndMean many thanks for the input. I'm turning your coment into several threads because otherwise it will be nearly impossible to converse about thes eeveral sub-suggestions made therein, feel free to comment in the respective threads

[Scienfitz]

via Kevin: Rename to CLAUDE.md + symlink for other tools

Consider making CLAUDE.md the canonical file (auto-loaded by Claude Code, supports subdirectory scoping). Symlink AGENTS.md and .github/copilot-instructions.md to it so all three tools share a single source of truth.

[Scienfitz]

@LeanAndMean not sure if I got your suggestions because that was already fully the idea of the AGENTS.md files. Afaik tools like claude or opencode alreayd recognize AGENTS.md (including auto-context-ingestion and subdirectory scoping) and do not really need CLAUDE.md or any other more specifically named file.

[Scienfitz]

via Kevin: Add a commands/validation section

The current file covers conventions but doesn't tell the agent how to validate its work. Add a commands section covering install, test (pytest --fast, pytest -k "test_name"), lint, typecheck, and tox environments (including tox -e mypy-py310 and tox -p for parallel).

[Scienfitz]

I like the idea maybe we could put that in or expand CONTROBUTING and link it in AGENTS

[Scienfitz]

via Kevin: Add an architecture-to-filepath mapping

Map each core domain concept to its file path (e.g. Campaign → baybe/campaign.py, SearchSpace → baybe/searchspace/). Gives agents "what is this" and "where to find it" in one pass.

[Scienfitz]

via Kevin: Distill key design principles for agents

Highlight the 2–3 principles agents are most likely to violate: comp-rep boundary, lazy imports ("non-negotiable"), and the serialization pattern. Complements the fuller treatment already in the file.

[Scienfitz]

via Kevin: Reduce root file token footprint

The root file is injected into every conversation. Consider trimming it (~270 lines → ~161) or moving depth into scoped subdirectory files to save token budget.

[Scienfitz]

I've already reduces the token footprint severly, they are currently at:

File	Lines	Words	Token estimate
AGENTS.md (root)	276	1,898	~2,500
docs/AGENTS.md	37	229	~300
examples/AGENTS.md	29	167	~230
tests/AGENTS.md	72	384	~550
Total	414	2,678	~3,600

Isnt that negligible already?

[Scienfitz]

via Kevin and Drew: Add concrete examples to anti-pattern rules

Rules like "prefer local fixtures" are easier for agents to follow with a one-liner showing the preferred pattern. Even minimal examples help.