Support running test262 using boa cli by ivankra · Pull Request #5234 · boa-dev/boa

ivankra · 2026-03-23T09:16:47Z

This Pull Request fixes/closes #5054

It changes the following:

Move tests/tester/src/exec/js262.rs to core/runtime/src/test262.rs so it can be reused by both cli and boa_tester
Add --test262-object flag to CLI
Add print() to CLI with --test262-object
Enable [[CanBlock]] by default in CLI to match boa_tester as well as d8/jsc/sm shells default behavior
Add --no-can-block flag to CLI to support Atomics CanBlockIsFalse tests (same flag name as d8 has)
Auto-detect .mjs files as modules in CLI (a small optional quality-of-life improvement; matches behavior of most other major JS engine shells)

These changes make it possible to reproduce boa_tester's test262 results using external test262 harnesses with boa cli binary.

core/runtime/src/test262.rs

github-actions · 2026-03-24T16:02:41Z

Test262 conformance changes

Test result	main count	PR count	difference
Total	52,963	52,963	0
Passed	50,735	50,735	0
Ignored	1,426	1,426	0
Failed	802	802	0
Panics	0	0	0
Conformance	95.79%	95.79%	0.00%

Tested main commit: f5e88de558e038f0ae675a012d59917a098f44b6
Tested PR commit: 85df98d49e4a6cf373c5aec1d66e1e90ac33af54
Compare commits: f5e88de...85df98d

ivankra · 2026-03-26T04:05:45Z

Fixed lint errors in lib.rs / test262.rs and rebased.

core/runtime/src/test262.rs

Move `$262` implementation from `tests/tester/src/exec/js262.rs` to `core/runtime/src/test262.rs` behind a new `test262` feature gate (pulls `bus` as optional dep). Add `annex-b` feature to boa_runtime forwarding to `boa_engine/annex-b` for `$262.IsHTMLDDA`. Both cli and boa_tester now use `boa_runtime/test262`, removing the tester's direct `bus` and `boa_gc` deps. CLI registers `$262` when `--test262-object` flag is passed. Switched $262.agent.monotonicNow() to context.clock().now().millis_since_epoch(). This change makes CLI more suitable for running test262 via an external harness such as eshost + test262-harness. The exact same `$262` object as used by the internal boa_tester can now be directly accessed in CLI. Existing `$boa` object doesn't expose some of the methods (like IsHTMLDDA, detachArrayBuffer and agent helpers) needed for full test262 support, and would have required users to maintain a custom shim mapping it to `$262`. Fixes boa-dev#5054

test262 tests use print() to signal async completion and errors. The tester has its own print() implementation to catch reported errors, but CLI should just print them, so let's alias it to console.log().

Treat files with .mjs extension as modules without requiring --module flag. Many other JavaScript shells do the same - d8, jsc, graaljs, xs, escargot. A small quality-of-life improvement that among other things simplifies configuration for an external test262 harness. The harness could simply stage module tests into *.mjs and forget about --module flag.

Make boa's cli consistent with the behaviour of other major JavaScript shells (d8, jsc, spidermonkey at least) as well as boa_tester by enabling [[CanBlock]] by default. Add --no-can-block flag to opt out, same as V8 (https://github.com/v8/v8/blob/main/src/d8/d8.cc#L6417). Needed for Atomics tests to pass when using an external test262 harness.

ivankra · 2026-03-29T16:51:02Z

Rebased again.

ivankra · 2026-03-29T18:41:45Z

I ran ./target/release/boa_tester run with default configuration (test262 commit: 72ec758c) - output is unchanged between f5e88de and 85df98d, so I think GitHub CI diff above is flaky:

Results (ECMAScript Next):
Total tests: 52963
Passed tests: 50745
Ignored tests: 1426
Failed tests: 792 (0 panics)
Conformance: 95.81%

Testing boa cli with my patches and my own test262 harness on the same test262 commit vs boa_tester I'm down to just a handful of diffs:

test/built-ins/Atomics/waitAsync/bigint/true-for-timeout.js: "AsyncTestFailure: Test timed out" vs OK
test/built-ins/Atomics/waitAsync/returns-result-object-value-is-promise-resolves-to-timed-out.js: "AsyncTestFailure: Test timed out" vs OK
test/built-ins/Atomics/waitAsync/true-for-timeout.js: "AsyncTestFailure: Test timed out" vs OK
test/language/import/import-defer/errors/syntax-error/import-defer-of-syntax-error-fails.js: "OK" vs FAIL
test/language/module-code/source-phase-import/import-source.js: "OK" vs FAIL
test/staging/sm/strict/8.7.2.js: "Test262Error: Expected SameValue(«false», «true») to be true" vs OK
test/staging/sm/strict/8.12.7-2.js: "Test262Error: Expected SameValue(«false», «true») to be true" vs OK
test/staging/sm/strict/10.4.2.js: "Test262Error: Expected SameValue(«false», «true») to be true" vs OK
test/staging/sm/strict/10.6.js: "Test262Error: Expected SameValue(«false», «true») to be true" vs OK
test/staging/sm/strict/12.10.1.js: "Test262Error: Expected SameValue(«false», «true») to be true" vs OK
test/staging/sm/strict/15.3.5.1.js: "Test262Error: Expected SameValue(«false», «true») to be true" vs OK
test/staging/sm/strict/15.3.5.2.js: "Test262Error: Expected SameValue(«false», «true») to be true" vs OK
test/staging/sm/strict/15.5.5.2.js: "Test262Error: Expected SameValue(«false», «true») to be true" vs OK
test/staging/source-phase-imports/import-source-source-text-module.js: "AsyncTestFailure: Promise should be rejected with SyntaxError Expected a SyntaxError to be thrown asynchronously but got TypeError" vs OK

test/staging/sm/strict/ I think would be better fixed upstream in test262 repo: tc39/test262#5008

jedel1043 · 2026-03-29T19:20:54Z

Nah the CI diff hasn't changed because actions haven't run haha

codecov · 2026-03-29T19:40:49Z

Codecov Report

❌ Patch coverage is 0% with 22 lines in your changes missing coverage. Please review.
✅ Project coverage is 59.68%. Comparing base (6ddc2b4) to head (85df98d).
⚠️ Report is 925 commits behind head on main.

Files with missing lines	Patch %	Lines
cli/src/main.rs	0.00%	12 Missing ⚠️
core/runtime/src/test262.rs	0.00%	10 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##             main    #5234       +/-   ##
===========================================
+ Coverage   47.24%   59.68%   +12.44%     
===========================================
  Files         476      589      +113     
  Lines       46892    63479    +16587     
===========================================
+ Hits        22154    37889    +15735     
- Misses      24738    25590      +852

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

ivankra requested a review from a team as a code owner March 23, 2026 09:16

github-actions bot added this to the v1.0.0 milestone Mar 23, 2026

ivankra mentioned this pull request Mar 23, 2026

Expose full $262 object in boa_cli #5054

Open

jedel1043 reviewed Mar 23, 2026

View reviewed changes

core/runtime/src/test262.rs Outdated Show resolved Hide resolved

ivankra force-pushed the test262 branch from c0b0380 to 4c52207 Compare March 23, 2026 10:37

ivankra force-pushed the test262 branch from 4c52207 to fb4249c Compare March 26, 2026 04:04

ivankra mentioned this pull request Mar 27, 2026

Add Boa to Test262-Harness #3431

Open

4 tasks

jedel1043 reviewed Mar 29, 2026

View reviewed changes

core/runtime/src/test262.rs Outdated Show resolved Hide resolved

ivankra force-pushed the test262 branch from fb4249c to 0a3a5f6 Compare March 29, 2026 16:47

ivankra added 4 commits March 30, 2026 00:48

cli: define print() for test262 compatibility

26a996a

test262 tests use print() to signal async completion and errors. The tester has its own print() implementation to catch reported errors, but CLI should just print them, so let's alias it to console.log().

ivankra force-pushed the test262 branch from 0a3a5f6 to 85df98d Compare March 29, 2026 16:50

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Support running test262 using boa cli#5234

Support running test262 using boa cli#5234
ivankra wants to merge 4 commits intoboa-dev:mainfrom
ivankra:test262

ivankra commented Mar 23, 2026 •

edited

Loading

Uh oh!

Uh oh!

github-actions bot commented Mar 24, 2026 •

edited

Loading

Uh oh!

ivankra commented Mar 26, 2026

Uh oh!

Uh oh!

ivankra commented Mar 29, 2026

Uh oh!

ivankra commented Mar 29, 2026 •

edited

Loading

Uh oh!

jedel1043 commented Mar 29, 2026

Uh oh!

codecov bot commented Mar 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

ivankra commented Mar 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Mar 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test262 conformance changes

Uh oh!

ivankra commented Mar 26, 2026

Uh oh!

Uh oh!

ivankra commented Mar 29, 2026

Uh oh!

ivankra commented Mar 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jedel1043 commented Mar 29, 2026

Uh oh!

codecov bot commented Mar 29, 2026

Codecov Report

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

ivankra commented Mar 23, 2026 •

edited

Loading

github-actions bot commented Mar 24, 2026 •

edited

Loading

ivankra commented Mar 29, 2026 •

edited

Loading