[pull] forks/amsterdam from ethereum:forks/amsterdam

.claude/commands/pytester.md

@@ -0,0 +1,36 @@
+# Pytester
+
+Guide for pytester-based plugin/CLI tests. Run before writing or modifying these tests.
+
+## Which execution mode to use
+
+- **`runpytest()`** — default. In-process, fast, full `RunResult` API (`assert_outcomes()`, `fnmatch_lines()`).
+- **`runpytest_subprocess()`** — use only when in-process causes state leakage (Pydantic cache pollution, global mutation in `pytest_configure`). Same `RunResult` API.
+- **Raw `subprocess.run()`** — never use alongside pytester. Use `runpytest_subprocess()` instead.
+
+Subprocess isolation masks bugs rather than fixing them. Prefer fixing the root cause and use subprocess as defense-in-depth.
+
+## Expected inner failures
+
+`runpytest_subprocess()` replays inner output to outer stdout (by design). Suppress with `capsys.readouterr()`:
+
+```python
+def test_expected_failure(pytester: Any, capsys: Any, pytestconfig: Any) -> None:
+    result = pytester.runpytest_subprocess(...)
+    capsys.readouterr()  # suppress inner failure bleed
+    assert result.ret != 0
+    output = "\n".join(result.outlines + result.errlines)
+    # conditional print for -s debugging
+    if pytestconfig.getoption("capture") == "no":
+        with capsys.disabled():
+            print(output)
+```
+
+## RunResult API
+
+Prefer `assert_outcomes()` and `fnmatch_lines()` over manual `any(... in line ...)` — better failure messages.
+
+- `result.ret` — exit code
+- `result.outlines` / `result.errlines` — line lists
+- `result.assert_outcomes(passed=N, failed=N)`
+- `result.stdout.fnmatch_lines(["*pattern*"])`

.github/workflows/hive-execute.yaml

@@ -0,0 +1,97 @@
+name: Hive Execute E2E
+
+on:
+  push:
+    branches:
+      - "forks/**"
+    paths:
+      - "packages/testing/src/execution_testing/cli/pytest_commands/plugins/execute/**"
+      - "packages/testing/src/execution_testing/rpc/**"
+      - ".github/workflows/hive-execute.yaml"
+      - ".github/actions/start-hive-dev/**"
+  pull_request:
+    paths:
+      - "packages/testing/src/execution_testing/cli/pytest_commands/plugins/execute/**"
+      - "packages/testing/src/execution_testing/rpc/**"
+      - ".github/workflows/hive-execute.yaml"
+      - ".github/actions/start-hive-dev/**"
+  workflow_dispatch:
+
+concurrency:
+  group: hive-execute-${{ github.workflow }}-${{ github.ref || github.run_id }}
+  cancel-in-progress: ${{ github.ref_name != github.event.repository.default_branch }}
+
+jobs:
+  cache-docker-images:
+    name: Cache Docker Images
+    runs-on: [self-hosted-ghr, size-l-x64]
+    steps:
+      - name: Checkout execution-specs
+        uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
+
+      - name: Cache Docker images
+        uses: ./.github/actions/cache-docker-images
+        with:
+          images: "docker.io/ethereum/client-go:latest docker.io/alpine:latest docker.io/library/golang:1-alpine"
+
+  test-execute-remote:
+    name: Execute Remote E2E
+    needs: cache-docker-images
+    runs-on: [self-hosted-ghr, size-l-x64]
+    steps:
+      - name: Checkout execution-specs
+        uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
+        with:
+          path: execution-specs
+          submodules: recursive
+
+      - name: Checkout Hive
+        uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
+        with:
+          repository: ethereum/hive
+          ref: master
+          path: hive
+
+      - name: Setup Go
+        uses: actions/setup-go@d35c59abb061a4a6fb18e82ac0862c26744d6ab5 # v5.5.0
+        with:
+          go-version: ">=1.24"
+          cache-dependency-path: hive/go.sum
+
+      - name: Install uv and python
+        uses: ./execution-specs/.github/actions/setup-uv
+        with:
+          cache-dependency-glob: "execution-specs/uv.lock"
+
+      - name: Setup environment
+        uses: ./execution-specs/.github/actions/setup-env
+
+      - name: Load cached Docker images
+        uses: ./execution-specs/.github/actions/load-docker-images
+
+      - name: Build Hive
+        run: |
+          cd hive
+          go build .
+
+      - name: Start Hive in dev mode
+        id: start-hive
+        uses: ./execution-specs/.github/actions/start-hive-dev
+        with:
+          clients: go-ethereum
+          client-file: execution-specs/.github/configs/hive/latest.yaml
+          hive-path: hive
+          timeout: "120"
+
+      - name: Run execute remote E2E tests
+        working-directory: execution-specs
+        env:
+          HIVE_SIMULATOR: ${{ steps.start-hive.outputs.hive-url }}
+        run: |
+          uv sync --all-extras
+          cd packages/testing
+          uv run pytest \
+            --basetemp="${{ runner.temp }}/pytest" \
+            -v \
+            -p execution_testing.cli.pytest_commands.plugins.concurrency \
+            src/execution_testing/cli/pytest_commands/plugins/execute/tests/test_execute_remote.py

CLAUDE.md

@@ -42,6 +42,7 @@ When reviewing PRs that implement or test EIPs:
 ## When to Use Skills
 
 - Writing or modifying tests → run `/write-test` first
+- Writing or modifying pytester-based plugin tests → run `/pytester` first
 - Filling test fixtures → run `/fill-tests` first
 - Implementing an EIP or modifying fork code in `src/` → run `/implement-eip` first
 - Modifying GitHub Actions workflows → run `/edit-workflow` first
@@ -53,6 +54,7 @@ When reviewing PRs that implement or test EIPs:
 ## Available Skills
 
 - `/write-test` — test writing patterns, fixtures, markers, bytecode helpers
+- `/pytester` — pytester execution modes, isolation, output handling for plugin tests
 - `/fill-tests` — `fill` CLI reference, flags, debugging, benchmark tests
 - `/implement-eip` — fork structure, import rules, adding opcodes/precompiles/tx types
 - `/edit-workflow` — GitHub Actions conventions and version pinning

docs/getting_started/code_standards_details.md

@@ -91,23 +91,17 @@ uvx pre-commit install
 
 For more information, see [Pre-commit Hooks Documentation](../dev/precommit.md).
 
-## Formatting and Line Length
+## Testing Framework Plugins with Pytester
 
-The Python code in @ethereum/execution-spec-tests is formatted with `ruff` with a line length of 100 characters.
+Use pytest's `pytester` fixture when writing tests for our pytest plugins and CLI commands.
 
-### Ignoring Bulk Change Commits
+`runpytest()` is the default. It runs the inner session in-process, is fast, and gives access to helpers like `assert_outcomes()` and `fnmatch_lines()`.
 
-The maximum line length was changed from 80 to 100 in Q2 2023. To ignore this bulk change commit in git blame output, use the `.git-blame-ignore-revs` file:
+`runpytest_subprocess()` runs the inner session in a separate process. Use it only when in-process mode causes state leakage (e.g., Pydantic `ModelMetaclass` cache pollution or global mutation in `pytest_configure`). Subprocess isolation masks these bugs rather than fixing them, so prefer fixing the root cause and use subprocess mode as defense-in-depth.
 
-```console
-git blame --ignore-revs-file .git-blame-ignore-revs docs/gen_test_case_reference.py
-```
-
-To use the revs file persistently with `git blame`:
+Don't use raw `subprocess.run()` in pytester-based tests. If you need process isolation, use `runpytest_subprocess()`.
 
-```console
-git config blame.ignoreRevsFile .git-blame-ignore-revs
-```
+Both methods return a `RunResult` with `.ret`, `.outlines`, `.errlines`, `assert_outcomes()`, and `fnmatch_lines()`. When the inner test is expected to fail, use `capsys.readouterr()` after `runpytest_subprocess()` to suppress the inner failure output that pytester replays to stdout.
 
 ## Building and Verifying Docs Locally
 

packages/testing/src/execution_testing/cli/pytest_commands/plugins/execute/rpc/chain_builder_eth_rpc.py

@@ -7,6 +7,7 @@
 from contextlib import AbstractContextManager
 from pathlib import Path
 from typing import Any, List, Sequence
+from urllib.parse import urlparse
 
 from filelock import FileLock
 
@@ -62,14 +63,15 @@ def __init__(
         )
         self.fork = fork
         self.engine_rpc = engine_rpc
+        parsed = urlparse(rpc_endpoint)
         self.block_building_lock = FileLock(
-            session_temp_folder / "chain_builder_fcu.lock"
+            session_temp_folder / f"chain_builder_fcu_{parsed.hostname}.lock"
         )
         self.get_payload_wait_time = get_payload_wait_time
         self.testing_rpc = testing_rpc
 
         # Send initial forkchoice updated only if we are the first worker
-        base_name = "eth_rpc_forkchoice_updated"
+        base_name = f"eth_rpc_forkchoice_updated_{parsed.hostname}"
         base_file = session_temp_folder / base_name
         base_error_file = session_temp_folder / f"{base_name}.err"
 

packages/testing/src/execution_testing/cli/pytest_commands/plugins/execute/rpc/remote.py

@@ -92,6 +92,16 @@ def pytest_addoption(parser: pytest.Parser) -> None:
             "only the JWT secret as a hex string."
         ),
     )
+    engine_rpc_group.addoption(
+        "--session-sync-folder",
+        required=False,
+        action="store",
+        default=None,
+        dest="session_sync_folder",
+        help=(
+            "Folder used to sync multiple instances of the execute command."
+        ),
+    )
 
 
 def pytest_configure(config: pytest.Config) -> None:
@@ -198,11 +208,14 @@ def eth_rpc(
     testing_rpc = None
     if use_testing_build_block:
         testing_rpc = TestingRPC(rpc_endpoint)
+    session_sync_folder = request.config.getoption("session_sync_folder")
     return ChainBuilderEthRPC(
         rpc_endpoint=rpc_endpoint,
         fork=session_fork,
         engine_rpc=engine_rpc,
-        session_temp_folder=session_temp_folder,
+        session_temp_folder=Path(session_sync_folder)
+        if session_sync_folder is not None
+        else session_temp_folder,
         get_payload_wait_time=get_payload_wait_time,
         transaction_wait_timeout=tx_wait_timeout,
         max_transactions_per_batch=max_transactions_per_batch,