From 8cbab2ce1958b8701cb97dd4fe12b5218e35da12 Mon Sep 17 00:00:00 2001 From: jlnav Date: Wed, 4 Mar 2026 14:42:04 -0600 Subject: [PATCH 1/4] initial commit - first draft --- AGENTS.md | 74 +++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 74 insertions(+) create mode 100644 AGENTS.md diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 000000000..96682f52d --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,74 @@ +Agent Contributor Guidelines and Information +============================================ + +Read the ``README.rst`` for an overview of libEnsemble. + +Repository Layout +----------------- + +- ``libensemble/`` - source code. +- ``/alloc_funcs`` - allocation functions. Policies for passing work between the manager and workers. +- ``/comms`` - modules and abstractions for communication between the manager and workers. +- ``/executors`` - an interface for launching executables, often simulations. +- ``/gen_classes`` - generators that adhere to the `gest-api` standard. + Recommended over entries from ``/gen_funcs`` that perform similar functionality. +- ``/gen_funcs`` - generator functions. Modules for producing points for simulations. +- ``/resources`` - classes and functions for managing compute resources for MPI tasks, libensemble workers. +- ``/sim_funcs`` - simulator functions. Modules for running simulations or performing experiments. +- ``/tests`` - tests. + - ``/functionality_tests`` primarily tests libEnsemble code only. + - ``/regression_tests`` tests libEnsemble code with external code. Often more closely resembles actual use-cases. + - ``/unit_tests`` tests for individual modules. +- ``/tools`` - tools. misc functions and classes to ease development. +- ``/utils`` - utilities. misc functions and classes used internally by multiple modules. +- ``ensemble.py`` - The primary interface for parameterizing and running libEnsemble. +- ``generators.py`` - base classes for generators that adhere to the `gest-api` standard. +- ``history.py`` - module for recording points that have been generated and simulation results. NumPy array. +- ``libE.py`` - libE main file. Previous primary interface for parameterizing and running libEnsemble. +- ``logger.py`` - logging configuration +- ``manager.py`` - module for maintaining the history array and passing points between the workers. +- ``message_numbers.py`` - constants that represent states of the ensemble. +- ``specs.py`` - Dataclasses for parameterizing the ensemble. +- ``worker.py`` - module for running generators and simulators. Communicates with the manager. +- ``version.py`` - version file. + +- ``.github/`` - GitHub actions. See ``.github/workflows/`` for the CI. +- ``.docs/`` - Documentation. Check here first for information before reading the source code. +- ``examples/`` - Symlinks to examples further inside the ``libensemble/`` directory. + +Other files in the root directory should be self-documenting. + +Familiarize yourself with ``libensemble/tests/regression_tests/test_1d_sampling.py`` for a simple example of the libEnsemble interface. + +General Guidelines +------------------ + +- If using a generator that adheres to the `gest-api` standard, use the ``start_only_persistent`` allocation function. +- An MPI distribution is not required for libEnsemble to run, but is required to use the ``MPIExecutor``. ``mpich`` is recommended. +- New tests are heavily encouraged for new features, bug fixes, or integrations. See ``libensemble/tests/regression_tests`` for examples. +- Never use destructive git commands unless explicitly requested. +- Code is in the ``black`` style. This should be enforced by ``pre-commit``. +- Read ``CONTRIBUTING.md`` for more information. + +Development Environment +----------------------- + +- ``pixi`` is the recommended environment manager for libEnsemble development. See ``pyproject.toml`` for the list +of dependencies and the available testing environments. + +- Enter the development environment with ``pixi shell -e dev``. This environment contains the most common dependencies for development and testing. + +- If ``pixi`` is not available or not preferred by the user, ``pip`` can be used instead, but the user will need to manually install other dependencies. + +- If committing, use ``pre-commit`` to ensure that code style and formatting are consistent. See ``.pre-commit-config.yaml`` for +the configuration and ``pyproject.toml`` for other configuration. + +Testing +------- + +- Run tests with the ``run-tests.py`` script. See ``libensemble/tests/run-tests.py`` for usage information. + +- Some tests require third party software to be installed. When developing a feature or fixing a bug the entire test suite may not necessarily need to be run, +since it will be on Github Actions. + +- Individual unit tests can be run with ``pytest``. From 334565e907dc1712cd45f8719f62f18ca1cfdb52 Mon Sep 17 00:00:00 2001 From: jlnav Date: Thu, 5 Mar 2026 11:29:36 -0600 Subject: [PATCH 2/4] additional clarifying and infrastructure details as suggestd by gemini and claude --- AGENTS.md | 76 +++++++++++++++++++++++++++++++++++-------------------- 1 file changed, 49 insertions(+), 27 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 96682f52d..af396d3a2 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -3,51 +3,70 @@ Agent Contributor Guidelines and Information Read the ``README.rst`` for an overview of libEnsemble. +- libEnsemble uses a manager-worker architecture. Points are generated by a generator and sent to a worker, which runs a simulator. +- The manager determines how and when points get passed to workers via an allocation function. + Repository Layout ----------------- -- ``libensemble/`` - source code. -- ``/alloc_funcs`` - allocation functions. Policies for passing work between the manager and workers. -- ``/comms`` - modules and abstractions for communication between the manager and workers. -- ``/executors`` - an interface for launching executables, often simulations. -- ``/gen_classes`` - generators that adhere to the `gest-api` standard. +- ``libensemble/`` - Source code. +- ``/alloc_funcs`` - Allocation functions. Policies for passing work between the manager and workers. +- ``/comms`` - Modules and abstractions for communication between the manager and workers. +- ``/executors`` - An interface for launching executables, often simulations. +- ``/gen_classes`` - Generators that adhere to the `gest-api` standard. Recommended over entries from ``/gen_funcs`` that perform similar functionality. -- ``/gen_funcs`` - generator functions. Modules for producing points for simulations. -- ``/resources`` - classes and functions for managing compute resources for MPI tasks, libensemble workers. -- ``/sim_funcs`` - simulator functions. Modules for running simulations or performing experiments. -- ``/tests`` - tests. - - ``/functionality_tests`` primarily tests libEnsemble code only. - - ``/regression_tests`` tests libEnsemble code with external code. Often more closely resembles actual use-cases. - - ``/unit_tests`` tests for individual modules. -- ``/tools`` - tools. misc functions and classes to ease development. -- ``/utils`` - utilities. misc functions and classes used internally by multiple modules. +- ``/gen_funcs`` - Generator functions. Modules for producing points for simulations. +- ``/resources`` - Classes and functions for managing compute resources for MPI tasks, libensemble workers. +- ``/sim_funcs`` - Simulator functions. Modules for running simulations or performing experiments. +- ``/tests`` - Tests. + - ``/functionality_tests`` - Primarily tests libEnsemble code only. + - ``/regression_tests`` - Tests libEnsemble code with external code. Often more closely resembles actual use-cases. + - ``/unit_tests`` - Tests for individual modules. +- ``/tools`` - Tools. Misc functions and classes to ease development. +- ``/utils`` - Utilities. Misc functions and classes used internally by multiple modules. - ``ensemble.py`` - The primary interface for parameterizing and running libEnsemble. -- ``generators.py`` - base classes for generators that adhere to the `gest-api` standard. -- ``history.py`` - module for recording points that have been generated and simulation results. NumPy array. +- ``generators.py`` - Base classes for generators that adhere to the `gest-api` standard. +- ``history.py`` - Module for recording points that have been generated and simulation results. NumPy array. - ``libE.py`` - libE main file. Previous primary interface for parameterizing and running libEnsemble. -- ``logger.py`` - logging configuration -- ``manager.py`` - module for maintaining the history array and passing points between the workers. -- ``message_numbers.py`` - constants that represent states of the ensemble. -- ``specs.py`` - Dataclasses for parameterizing the ensemble. -- ``worker.py`` - module for running generators and simulators. Communicates with the manager. -- ``version.py`` - version file. +- ``logger.py`` - Logging configuration +- ``manager.py`` - Module for maintaining the history array and passing points between the workers. +- ``message_numbers.py`` - Constants that represent states of the ensemble. +- ``specs.py`` - Dataclasses for parameterizing the ensemble. Most importantly, contains ``LibeSpecs, SimSpecs, GenSpecs``. +- ``worker.py`` - Module for running generators and simulators. Communicates with the manager. +- ``version.py`` - Version file. - ``.github/`` - GitHub actions. See ``.github/workflows/`` for the CI. -- ``.docs/`` - Documentation. Check here first for information before reading the source code. -- ``examples/`` - Symlinks to examples further inside the ``libensemble/`` directory. +- ``docs/`` - Documentation. Check here first for information before reading the source code. +- ``examples/`` - The ``*_funcs`` and ``calling_scripts`` directories contain symlinks to examples further in the source code. +- ``/libE_submission_scripts`` - Example scripts for submitting libEnsemble jobs to HPC systems. +- ``/tutorials`` - Tutorials on how to use libEnsemble. +- ``pyproject.toml`` - Project configuration file. Contains information about the project and its dependencies. Other files in the root directory should be self-documenting. Familiarize yourself with ``libensemble/tests/regression_tests/test_1d_sampling.py`` for a simple example of the libEnsemble interface. +Information about Generators +---------------------------- + +- Generators are functions or objects that produce points for simulations. +- The History array is a numpy structured array that stores points that have been generated and simulation results. Its fields match ``sim_specs/gen_specs["out"]`` or ``vocs`` attributes. +- Prior to libEnsemble v1.6.0, generators were plain functions. They often ran in "persistent" mode, meaning they executed in a long-running loop, sending and receiving points to and from the manager until the ensemble was complete. +- A ``gest-api`` or "standardized" generator is a class that at a minimum implements ``suggest`` and ``ingest`` methods, and is parameterized by a ``vocs``. +- See ``libensemble/generators.py`` for more information about the ``gest-api`` standard. + General Guidelines ------------------ -- If using a generator that adheres to the `gest-api` standard, use the ``start_only_persistent`` allocation function. +- If using classic ``sim_specs`` and ``gen_specs``, then ensure that ``sim_specs["out"]`` and ``gen_specs["in"]`` field names match, and vice-versa. +- As-of libEnsemble v1.6.0, ``SimSpecs`` and ``GenSpecs`` can also be parameterized by a ``vocs`` object, imported from ``gest_api.vocs`` (NOT xopt.vocs). +- ``VOCS`` contains variables, objectives, constraints, and other settings that define the problem. See ``libensemble/tests/regression_tests/test_xopt_EI.py`` for an example of how to use it. +- If using a generator that adheres to the ``gest-api`` standard, or a classic persistent generator, use the ``start_only_persistent`` allocation function. - An MPI distribution is not required for libEnsemble to run, but is required to use the ``MPIExecutor``. ``mpich`` is recommended. - New tests are heavily encouraged for new features, bug fixes, or integrations. See ``libensemble/tests/regression_tests`` for examples. - Never use destructive git commands unless explicitly requested. - Code is in the ``black`` style. This should be enforced by ``pre-commit``. +- When writing new code, prefer the ``LibeSpecs``, ``SimSpecs``, and ``GenSpecs`` dataclasses over the classic ``sim_specs`` and ``gen_specs`` bare dictionaries. - Read ``CONTRIBUTING.md`` for more information. Development Environment @@ -57,6 +76,7 @@ Development Environment of dependencies and the available testing environments. - Enter the development environment with ``pixi shell -e dev``. This environment contains the most common dependencies for development and testing. +- For one-off commands, use ``pixi run -e dev``. This will run a single command in the development environment. - If ``pixi`` is not available or not preferred by the user, ``pip`` can be used instead, but the user will need to manually install other dependencies. @@ -66,9 +86,11 @@ the configuration and ``pyproject.toml`` for other configuration. Testing ------- -- Run tests with the ``run-tests.py`` script. See ``libensemble/tests/run-tests.py`` for usage information. +- Run tests with the ``run-tests.py`` script: ``python libensemble/tests/run-tests.py``. See ``libensemble/tests/run-tests.py`` for usage information. - Some tests require third party software to be installed. When developing a feature or fixing a bug the entire test suite may not necessarily need to be run, since it will be on Github Actions. -- Individual unit tests can be run with ``pytest``. +- Individual unit tests can be run with ``pixi run -e dev pytest path/to/test_file``. +- A libEnsemble run typically outputs an ``ensemble.log`` and ``libE_stats.txt`` file in the working directory. Check these files for tracebacks or run statistics. +- An "ensemble" or "workflow" directory may also be created, often containing per-simulation output directories From 72ed72bddb15124da3fdfdb521db57b316d89f01 Mon Sep 17 00:00:00 2001 From: jlnav Date: Thu, 5 Mar 2026 15:02:29 -0600 Subject: [PATCH 3/4] fix typo --- docs/examples/gest_api.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/examples/gest_api.rst b/docs/examples/gest_api.rst index e68180665..077355b77 100644 --- a/docs/examples/gest_api.rst +++ b/docs/examples/gest_api.rst @@ -98,7 +98,7 @@ Examples: `Grid Sampling`_ -`Ax Multi-fideltiy`_ +`Ax Multi-fidelity`_ .. _gest_api: https://github.com/campa-consortium/gest-api .. _gpcam: https://gpcam.lbl.gov/ @@ -110,4 +110,4 @@ Examples: .. _Grid Sampling: https://github.com/Libensemble/libensemble/blob/develop/libensemble/tests/regression_tests/test_optimas_grid_sample.py -.. _Ax Multi-fideltiy: https://github.com/Libensemble/libensemble/blob/develop/libensemble/tests/regression_tests/test_optimas_ax_mf.py +.. _Ax Multi-fidelity: https://github.com/Libensemble/libensemble/blob/develop/libensemble/tests/regression_tests/test_optimas_ax_mf.py From 5609ac1c55a05d121f33f5e95b638c9b1b2a111f Mon Sep 17 00:00:00 2001 From: jlnav Date: Fri, 6 Mar 2026 08:48:48 -0600 Subject: [PATCH 4/4] additional rearranging and clarifications --- AGENTS.md | 29 ++++++++++++++--------------- 1 file changed, 14 insertions(+), 15 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index af396d3a2..d3e3bd150 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -5,6 +5,7 @@ Read the ``README.rst`` for an overview of libEnsemble. - libEnsemble uses a manager-worker architecture. Points are generated by a generator and sent to a worker, which runs a simulator. - The manager determines how and when points get passed to workers via an allocation function. +- See ``libensemble/tests/regression_tests/test_1d_sampling.py`` for a simple example of the libEnsemble interface. Repository Layout ----------------- @@ -28,7 +29,7 @@ Repository Layout - ``generators.py`` - Base classes for generators that adhere to the `gest-api` standard. - ``history.py`` - Module for recording points that have been generated and simulation results. NumPy array. - ``libE.py`` - libE main file. Previous primary interface for parameterizing and running libEnsemble. -- ``logger.py`` - Logging configuration +- ``logger.py`` - Logging configuration. - ``manager.py`` - Module for maintaining the history array and passing points between the workers. - ``message_numbers.py`` - Constants that represent states of the ensemble. - ``specs.py`` - Dataclasses for parameterizing the ensemble. Most importantly, contains ``LibeSpecs, SimSpecs, GenSpecs``. @@ -44,42 +45,42 @@ Repository Layout Other files in the root directory should be self-documenting. -Familiarize yourself with ``libensemble/tests/regression_tests/test_1d_sampling.py`` for a simple example of the libEnsemble interface. - Information about Generators ---------------------------- - Generators are functions or objects that produce points for simulations. -- The History array is a numpy structured array that stores points that have been generated and simulation results. Its fields match ``sim_specs/gen_specs["out"]`` or ``vocs`` attributes. -- Prior to libEnsemble v1.6.0, generators were plain functions. They often ran in "persistent" mode, meaning they executed in a long-running loop, sending and receiving points to and from the manager until the ensemble was complete. +- The History array is a NumPy structured array that stores points that have been generated and simulation results. +Its fields match ``sim_specs/gen_specs["out"]`` or ``vocs`` attributes, plus additional reserved fields for metadata. +- Prior to libEnsemble v1.6.0, generators were plain functions. They often ran in "persistent" mode, meaning they executed in a +long-running loop, sending and receiving points to and from the manager until the ensemble was complete. - A ``gest-api`` or "standardized" generator is a class that at a minimum implements ``suggest`` and ``ingest`` methods, and is parameterized by a ``vocs``. - See ``libensemble/generators.py`` for more information about the ``gest-api`` standard. +- If using a generator that adheres to the ``gest-api`` standard, or a classic persistent generator, use the ``start_only_persistent`` allocation function. +- Generators are often used for simple sampling, optimization, calibration, uncertainty quantification, and other simulation-based tasks. General Guidelines ------------------ - If using classic ``sim_specs`` and ``gen_specs``, then ensure that ``sim_specs["out"]`` and ``gen_specs["in"]`` field names match, and vice-versa. - As-of libEnsemble v1.6.0, ``SimSpecs`` and ``GenSpecs`` can also be parameterized by a ``vocs`` object, imported from ``gest_api.vocs`` (NOT xopt.vocs). -- ``VOCS`` contains variables, objectives, constraints, and other settings that define the problem. See ``libensemble/tests/regression_tests/test_xopt_EI.py`` for an example of how to use it. -- If using a generator that adheres to the ``gest-api`` standard, or a classic persistent generator, use the ``start_only_persistent`` allocation function. +- ``VOCS`` contains variables, objectives, constraints, and other settings that define the problem. +See ``libensemble/tests/regression_tests/test_xopt_EI.py`` for an example of how to use it. - An MPI distribution is not required for libEnsemble to run, but is required to use the ``MPIExecutor``. ``mpich`` is recommended. - New tests are heavily encouraged for new features, bug fixes, or integrations. See ``libensemble/tests/regression_tests`` for examples. - Never use destructive git commands unless explicitly requested. - Code is in the ``black`` style. This should be enforced by ``pre-commit``. - When writing new code, prefer the ``LibeSpecs``, ``SimSpecs``, and ``GenSpecs`` dataclasses over the classic ``sim_specs`` and ``gen_specs`` bare dictionaries. - Read ``CONTRIBUTING.md`` for more information. +- The external ``libE-community-examples`` repository contains past use-cases, generators, and other examples. Development Environment ----------------------- - ``pixi`` is the recommended environment manager for libEnsemble development. See ``pyproject.toml`` for the list of dependencies and the available testing environments. - - Enter the development environment with ``pixi shell -e dev``. This environment contains the most common dependencies for development and testing. - For one-off commands, use ``pixi run -e dev``. This will run a single command in the development environment. - -- If ``pixi`` is not available or not preferred by the user, ``pip`` can be used instead, but the user will need to manually install other dependencies. - +- If ``pixi`` is not available or not preferred by the user, ``pip install -e .`` can be used instead. Other dependencies may need to be installed manually. - If committing, use ``pre-commit`` to ensure that code style and formatting are consistent. See ``.pre-commit-config.yaml`` for the configuration and ``pyproject.toml`` for other configuration. @@ -87,10 +88,8 @@ Testing ------- - Run tests with the ``run-tests.py`` script: ``python libensemble/tests/run-tests.py``. See ``libensemble/tests/run-tests.py`` for usage information. - -- Some tests require third party software to be installed. When developing a feature or fixing a bug the entire test suite may not necessarily need to be run, -since it will be on Github Actions. - +- Some tests require third party software to be installed. When developing a feature or fixing a bug, since the entire test suite will be run on Github Actions, +for local development running individual tests is sufficient. - Individual unit tests can be run with ``pixi run -e dev pytest path/to/test_file``. - A libEnsemble run typically outputs an ``ensemble.log`` and ``libE_stats.txt`` file in the working directory. Check these files for tracebacks or run statistics. - An "ensemble" or "workflow" directory may also be created, often containing per-simulation output directories