Skip to content

ci: fix Windows workspace cleanup and fail-fast for git operations#3694

Draft
Leiyks wants to merge 2 commits intomasterfrom
leiyks/fix-windows-job-flakiness
Draft

ci: fix Windows workspace cleanup and fail-fast for git operations#3694
Leiyks wants to merge 2 commits intomasterfrom
leiyks/fix-windows-job-flakiness

Conversation

@Leiyks
Copy link
Contributor

@Leiyks Leiyks commented Mar 6, 2026

Problem

Windows CI jobs (compile extension windows, windows test_c) were intermittently failing with exit code 10 from phpize.bat — specifically the error Must be run from the root of the extension source.

Root cause was two compounding bugs introduced in #3634 when switching to GIT_STRATEGY: none + manual clone:

Bug 1 — Workspace cleanup fails silently, leaving leftover content

First attempt used cmd.exe "for /d" loop which skips directory entries while iterating (deletes and enumerates in the same pass — well-known Windows antipattern). Leftover artifact dirs from prior runs (extensions_x86_64, etc.) survived.

Second attempt switched to Get-ChildItem | Remove-Item -Recurse, which throws "mismatch between the tag specified in the request and the tag present in the reparse point" on PowerShell 5.1 (Windows Server 2019 default) when the workspace contains Windows junction points (created by switch-php, e.g. /php <<===>> /php-nts) or NTFS symlinks (from core.symlinks=true git clone). The entire cleanup silently fails, leaving the full previous repo tree in place.

Fix: navigate to the parent directory and use cmd.exe "rd /s /q" on the whole workspace, then recreate it empty. cmd.exe rd correctly removes junction entries without following them into their targets, avoiding the reparse point issue entirely.

Bug 2 — $PSNativeCommandUseErrorActionPreference is a no-op on PS 5.1

The runner uses the Shell (powershell) executor = Windows PowerShell 5.1 (default on Windows Server 2019). $PSNativeCommandUseErrorActionPreference was introduced in PowerShell 7.3 and is silently ignored on 5.1. So when git clone failed (non-empty workspace), the script continued without source code, started the Docker container with an empty mount, and phpize.bat failed.

Fix: remove $PSNativeCommandUseErrorActionPreference (misleading no-op) and add explicit $LASTEXITCODE checks after git clone, git checkout, and git submodule update — the only reliable fail-fast mechanism for native commands on PS 5.1.

Leiyks added 2 commits March 6, 2026 17:06
The cmd.exe "for /d" loop used to clean the workspace skips directory
entries during deletion (enumerates and deletes in the same pass), leaving
artifact output dirs from previous runs. When git clone then fails because
the workspace isn't empty, $PSNativeCommandUseErrorActionPreference = $true
is silently ignored on Windows PowerShell 5.1 (requires PS 7.3+), so the
script continues without source code and phpize.bat fails with exit 10.

Fixes:
- Replace cmd.exe cleanup loop with PowerShell-native Get-ChildItem | Remove-Item
  which handles each entry independently and tolerates locked files
- Add WARNING log line if any items could not be removed (aids debugging)
- Remove $PSNativeCommandUseErrorActionPreference (no-op on PS 5.1)
- Add explicit $LASTEXITCODE checks after git clone, checkout, and submodule init

Applied to both generate-package.php (compile extension windows) and
generate-tracer.php (windows test_c).
… cleanup

PowerShell 5.1's Remove-Item -Recurse throws "mismatch between the tag
specified in the request and the tag present in the reparse point" when
the workspace contains Windows junction points (created by switch-php,
e.g. /php <<===>> /php-nts) or NTFS symlinks (from core.symlinks=true
git clone). This caused the entire cleanup to fail silently, leaving
the full previous repo tree in place and making git clone fail again.

Fix: navigate to the parent directory and run cmd.exe "rd /s /q" on
the whole workspace directory. cmd.exe rd removes junction entries
without following them into their targets, avoiding the reparse point
issue entirely. The directory is then recreated empty before returning.
@datadog-official
Copy link

datadog-official bot commented Mar 6, 2026

⚠️ Tests

Fix all issues with BitsAI or with Cursor

⚠️ Warnings

🧪 1028 Tests failed

testSearchPhpBinaries from integration.DDTrace\Tests\Integration\PHPInstallerTest (Datadog) (Fix with Cursor)
DDTrace\Tests\Integration\PHPInstallerTest::testSearchPhpBinaries
Test code or tested code printed unexpected output: Searching for available php binaries, this operation might take a while.

phpvfscomposer://tests/vendor/phpunit/phpunit/phpunit:106
testSimplePushAndProcess from laravel-58-test.DDTrace\Tests\Integrations\Laravel\V5_8\QueueTest (Datadog) (Fix with Cursor)
Risky Test
phpvfscomposer://tests/vendor/phpunit/phpunit/phpunit:60
testSimplePushAndProcess from laravel-8x-test.DDTrace\Tests\Integrations\Laravel\V8_x\QueueTest (Datadog) (Fix with Cursor)
DDTrace\Tests\Integrations\Laravel\V8_x\QueueTest::testSimplePushAndProcess
Test code or tested code printed unexpected output: spanLinksTraceId: 69ab0fef000000005fb77d468e4d9b26
tid: 69ab0fef00000000
hexProcessTraceId: 5fb77d468e4d9b26
hexProcessSpanId: b54adb3826c58c98
processTraceId: 6897119096329837350
processSpanId: 13063494703356677272
View all

ℹ️ Info

❄️ No new flaky tests detected

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: f068d19 | Docs | Datadog PR Page | Was this helpful? React with 👍/👎 or give us feedback!

@codecov-commenter
Copy link

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 62.30%. Comparing base (b33ea57) to head (f068d19).

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #3694      +/-   ##
==========================================
- Coverage   62.32%   62.30%   -0.02%     
==========================================
  Files         142      142              
  Lines       13586    13586              
  Branches     1775     1775              
==========================================
- Hits         8467     8465       -2     
- Misses       4311     4314       +3     
+ Partials      808      807       -1     

see 1 file with indirect coverage changes


Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b33ea57...f068d19. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants