CI failures: 20180815

[joyeecheung]

Failures in the latest 30 CI runs that failed more than 1 PRs (I only went with 30 to compare with #12)

JSTest Failure

Reason	parallel/test-cluster-master-error
Type	JS_TEST_FAILURE
Failed PR	10 (nodejs/node#22293, nodejs/node#22307, nodejs/node#22310, nodejs/node#22306, nodejs/node#22249, nodejs/node#22320, nodejs/node#22318, nodejs/node#21611, nodejs/node#22128, nodejs/node#22269)
Appeared	test-joyent-alpine38_container-x64-1, test-softlayer-alpine38_container-x64-1
First CI	https://ci.nodejs.org/job/node-test-pull-request/16434/
Last CI	https://ci.nodejs.org/job/node-test-pull-request/16450/

Example

not ok 649 parallel/test-cluster-master-error
  ---
  duration_ms: 120.51
  severity: fail
  exitcode: -15
  stack: |-
    timeout
  ...

---

Reason	parallel/test-cluster-master-kill
Type	JS_TEST_FAILURE
Failed PR	10 (nodejs/node#22293, nodejs/node#22307, nodejs/node#22310, nodejs/node#22306, nodejs/node#22249, nodejs/node#22320, nodejs/node#22318, nodejs/node#21611, nodejs/node#22128, nodejs/node#22269)
Appeared	test-joyent-alpine38_container-x64-1, test-softlayer-alpine38_container-x64-1
First CI	https://ci.nodejs.org/job/node-test-pull-request/16434/
Last CI	https://ci.nodejs.org/job/node-test-pull-request/16450/

Example

not ok 650 parallel/test-cluster-master-kill
  ---
  duration_ms: 120.99
  severity: fail
  exitcode: -15
  stack: |-
    timeout
  ...

---

Reason	addons/stringbytes-external-exceed-max/test-stringbytes-external-exceed-max-by-1-binary
Type	JS_TEST_FAILURE
Failed PR	9 (nodejs/node#22074, nodejs/node#22106, nodejs/node#22266, nodejs/node#22300, nodejs/node#22258, nodejs/node#22284, nodejs/node#22295, nodejs/node#22293, nodejs/node#22320)
Appeared	test-digitalocean-freebsd11-x64-2, test-digitalocean-freebsd11-x64-1
First CI	https://ci.nodejs.org/job/node-test-pull-request/16418/
Last CI	https://ci.nodejs.org/job/node-test-pull-request/16458/

Example

not ok 1904 addons/stringbytes-external-exceed-max/test-stringbytes-external-exceed-max-by-1-binary
  ---
  duration_ms: 35.394
  severity: crashed
  stack: |-
    oh no!
    exit code: CRASHED (Signal: 9)
  ...

---

Reason	async-hooks/test-callback-error
Type	JS_TEST_FAILURE
Failed PR	7 (nodejs/node#22300, nodejs/node#22273, nodejs/node#22310, nodejs/node#22249, nodejs/node#22306, nodejs/node#22318, nodejs/node#22269)
Appeared	test-digitalocean-fedora27-x64-1
First CI	https://ci.nodejs.org/job/node-test-pull-request/16426/
Last CI	https://ci.nodejs.org/job/node-test-pull-request/16450/

Example

not ok 2052 async-hooks/test-callback-error
  ---
  duration_ms: 19.946
  severity: fail
  exitcode: 1
  stack: |-
    start case 1
    end case 1: 1546.396ms
    start case 2
    end case 2: 1719.529ms
    start case 3
    end case 3: 75.569ms
    Error: test_callback_abort
        at ActivityCollector.initHooks.oninit.common.mustCall (/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/test/async-hooks/test-callback-error.js:27:45)
        at ActivityCollector.oninit (/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/test/common/index.js:460:15)
        at ActivityCollector._init (/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/test/async-hooks/init-hooks.js:193:10)
        at emitInitNative (internal/async_hooks.js:137:43)
        at emitInitScript (internal/async_hooks.js:336:3)
        at new AsyncResource (async_hooks.js:163:5)
        at Object.<anonymous> (/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/test/async-hooks/test-callback-error.js:29:5)
        at Module._compile (internal/modules/cjs/loader.js:689:30)
        at Object.Module._extensions..js (internal/modules/cjs/loader.js:700:10)
        at Module.load (internal/modules/cjs/loader.js:599:32)
     1: 0x8bba80 node::Abort() [/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node]
     2: 0x9402e9  [/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node]
     3: 0xb62e79  [/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node]
     4: 0xb63d14 v8::internal::Builtin_HandleApiCall(int, v8::internal::Object**, v8::internal::Isolate*) [/home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/node]
     5: 0x3beaa35dc01d 
    
  ...

---

Reason	parallel/test-tls-passphrase
Type	JS_TEST_FAILURE
Failed PR	4 (nodejs/node#22320, nodejs/node#22318, nodejs/node#22330, nodejs/node#22128)
Appeared	test-digitalocean-ubuntu1604_sharedlibs_container-x64-6, test-softlayer-ubuntu1604_sharedlibs_container-x64-5, test-joyent-ubuntu1604_sharedlibs_container-x64-2
First CI	https://ci.nodejs.org/job/node-test-pull-request/16444/
Last CI	https://ci.nodejs.org/job/node-test-pull-request/16460/

Example

not ok 1798 parallel/test-tls-passphrase
  ---
  duration_ms: 0.391
  severity: fail
  exitcode: 1
  stack: |-
    assert.js:650
        throw actual;
        ^
    
    Error: error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt
        at Object.createSecureContext (_tls_common.js:144:17)
        at Object.connect (_tls_wrap.js:1136:48)
        at /home/iojs/build/workspace/node-test-commit-linux-containered/nodes/ubuntu1604_sharedlibs_openssl110_x64/test/parallel/test-tls-passphrase.js:228:7
        at getActual (assert.js:563:5)
        at Function.throws (assert.js:680:24)
        at Object.<anonymous> (/home/iojs/build/workspace/node-test-commit-linux-containered/nodes/ubuntu1604_sharedlibs_openssl110_x64/test/parallel/test-tls-passphrase.js:227:8)
        at Module._compile (internal/modules/cjs/loader.js:689:30)
        at Object.Module._extensions..js (internal/modules/cjs/loader.js:700:10)
        at Module.load (internal/modules/cjs/loader.js:599:32)
        at tryModuleLoad (internal/modules/cjs/loader.js:538:12)
  ...

---

Reason	async-hooks/test-statwatcher
Type	JS_TEST_FAILURE
Failed PR	3 (nodejs/node#22242, nodejs/node#21611, nodejs/node#22302)
Appeared	test-azure_msft-win2016-x64-4
First CI	https://ci.nodejs.org/job/node-test-pull-request/16424/
Last CI	https://ci.nodejs.org/job/node-test-pull-request/16451/

Example

not ok 12 async-hooks/test-statwatcher
  ---
  duration_ms: 0.463
  severity: fail
  exitcode: 1
  stack: |-
    assert.js:84
      throw new AssertionError(obj);
      ^
    
    AssertionError [ERR_ASSERTION]: Checking invocations at stage "watcher1: when unwatched second file":
         Called "before" 2 time(s), but expected 1 invocation(s).
        at checkHook (c:\workspace\node-test-binary-windows\test\async-hooks\hook-checks.js:51:14)
        at Array.forEach (<anonymous>)
        at checkInvocations (c:\workspace\node-test-binary-windows\test\async-hooks\hook-checks.js:28:62)
        at Immediate.setImmediate (c:\workspace\node-test-binary-windows\test\async-hooks\test-statwatcher.js:65:9)
        at runCallback (timers.js:661:18)
        at tryOnImmediate (timers.js:632:5)
        at processImmediate (timers.js:614:5)
  ...

---

Reason	parallel/test-crypto-scrypt
Type	JS_TEST_FAILURE
Failed PR	3 (nodejs/node#22318, nodejs/node#22330, nodejs/node#22128)
Appeared	test-digitalocean-ubuntu1604_sharedlibs_container-x64-6, test-softlayer-ubuntu1604_sharedlibs_container-x64-5
First CI	https://ci.nodejs.org/job/node-test-pull-request/16446/
Last CI	https://ci.nodejs.org/job/node-test-pull-request/16460/

Example

not ok 377 parallel/test-crypto-scrypt
  ---
  duration_ms: 2.480
  severity: fail
  exitcode: 1
  stack: |-
    /home/iojs/build/workspace/node-test-commit-linux-containered/nodes/ubuntu1604_sharedlibs_openssl110_x64/test/common/index.js:735
            throw new assert.AssertionError({
            ^
    
    AssertionError [ERR_ASSERTION]: Expected inputs to be strictly equal:
    + actual - expected ... Lines skipped
    
      Comparison {
    +   message: 'error:060B50AC:digital envelope routines:EVP_PBE_scrypt:memory limit exceeded',
    -   code: 'ERR_CRYPTO_SCRYPT_INVALID_PARAMETER',
    -   message: 'Invalid scrypt parameter',
        type: [Function: Error] {
    ...

---

Reason	parallel/test-gc-http-client
Type	JS_TEST_FAILURE
Failed PR	2 (nodejs/node#22074, nodejs/node#22152)
Appeared	test-digitalocean-ubuntu1604_sharedlibs_container-x64-10, test-digitalocean-freebsd11-x64-1, test-digitalocean-ubuntu1604_sharedlibs_container-x64-9
First CI	https://ci.nodejs.org/job/node-test-pull-request/16418/
Last CI	https://ci.nodejs.org/job/node-test-pull-request/16420/

Example

not ok 759 parallel/test-gc-http-client
  ---
  duration_ms: 480.47
  severity: fail
  exitcode: -15
  stack: |-
    timeout
    We should do 500 requests
    Done: 0/500
    Collected: 0/11
    Done: 0/500
    Collected: 0/22
    Done: 0/500
    Collected: 0/33
    Done: 0/500
    Collected: 0/44
    Done: 11/500
    Collected: 0/55
    Done: 22/500
    Collected: 0/66
    Done: 33/500
    Collected: 11/77
    Done: 44/500
    Collected: 22/88
    Done: 55/500
    Collected: 33/99
    Done: 66/500
    Collected: 44/110
    Done: 77/500
    Collected: 55/121
    Done: 88/500
    Collected: 66/132
    Done: 99/500
    Collected: 77/143
    Done: 110/500
    Collected: 88/154
    Done: 121/500
    Collected: 99/165
    Done: 132/500
    Collected: 110/176
    Done: 143/500
    Collected: 121/187
    Done: 154/500
    Collected: 132/198
    Done: 165/500
    Collected: 143/209
    Done: 176/500
    Collected: 154/220
    Done: 187/500
    Collected: 165/231
    Done: 198/500
    Collected: 176/242
    Done: 209/500
    Collected: 187/253
    Done: 220/500
    Collected: 198/264
    Done: 231/500
    Collected: 209/275
    Done: 242/500
    Collected: 220/286
    Done: 253/500
    Collected: 231/297
    Done: 264/500
    Collected: 242/308
    Done: 275/500
    Collected: 253/319
    Done: 286/500
    Collected: 264/330
    Done: 297/500
    Collected: 275/341
    Done: 308/500
    Collected: 286/352
    Done: 319/500
    Collected: 297/363
    Done: 330/500
    Collected: 308/374
    Done: 341/500
    Collected: 319/385
    Done: 352/500
    Collected: 330/396
    Done: 363/500
    Collected: 341/407
    Done: 374/500
    Collected: 352/418
    Done: 385/500
    Collected: 363/429
    Done: 396/500
    Collected: 374/440
    Done: 407/500
    Collected: 385/451
    Done: 418/500
    Collected: 396/462
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
    in cb
  ...

---

Git Failure

Reason	Changes not staged for commit:
Type	GIT_FAILURE
Failed PR	13 (nodejs/node#22106, nodejs/node#22242, nodejs/node#22300, nodejs/node#22258, nodejs/node#22284, nodejs/node#22273, nodejs/node#22301, nodejs/node#22293, nodejs/node#22307, nodejs/node#22310, nodejs/node#22269, nodejs/node#22318, nodejs/node#22330)
Appeared	test-packetnet-ubuntu1604-arm64-2, test-softlayer-ubuntu1604_sharedlibs_container-x64-2, test-softlayer-ubuntu1604_sharedlibs_container-x64-5, test-softlayer-ubuntu1604_sharedlibs_container-x64-1, test-linuxonecc-rhel72-s390x-2, test-packetnet-centos7-arm64-2, test-joyent-smartos16-x64-1, test-digitalocean-ubuntu1604_sharedlibs_container-x64-1, test-packetnet-centos7-arm64-1, test-linuxonecc-rhel72-s390x-3, test-digitalocean-ubuntu1604_sharedlibs_container-x64-3
First CI	https://ci.nodejs.org/job/node-test-pull-request/16422/
Last CI	https://ci.nodejs.org/job/node-test-pull-request/16461/

Example

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

	modified:   deps/v8/third_party/jinja2/LICENSE

Untracked files:
  (use "git add <file>..." to include in what will be committed)

	build/

no changes added to commit

---

Build Failure

Reason	fatal: Couldn't find remote ref refs/heads/v4.x-staging
Type	BUILD_FAILURE
Failed PR	3 (nodejs/node#22330, nodejs/node#22320, nodejs/node#22318)
Appeared	test-requireio_chrislea-debian9-armv6l_pi1p-1, test-requireio_mininodes-debian9-armv7l_pi2-1, test-requireio_mininodes-debian9-armv6l_pi1p-1, test-requireio_bengl-debian9-armv6l_pi1p-1, test-requireio_continuationlabs-debian9-armv6l_pi1p-1, test-requireio_svincent-debian9-armv7l_pi2-1, test-requireio_mcollina-debian9-armv7l_pi2-1, test-requireio_svincent-debian9-armv7l_pi2-2, test-requireio_joeyvandijk-debian9-armv7l_pi2-1, test-requireio_jasnell-debian9-armv7l_pi2-1
First CI	https://ci.nodejs.org/job/node-test-pull-request/16457/
Last CI	https://ci.nodejs.org/job/node-test-pull-request/16459/

Example

fatal: Couldn't find remote ref refs/heads/v4.x-staging
fatal: The remote end hung up unexpectedly

---

[joyeecheung]

Progress of the recent failures:

• parallel/test-cluster-master-error and parallel/test-cluster-master-kill
  • Being tracked in Alpine 3.8 cluster failures node#22308, the containers are being rebuilt
• addons/stringbytes-external-exceed-max/test-stringbytes-external-exceed-max-by-1-binary
  • Being temporarily fixed by test: call gc() explicitly to avoid OOM node#22301 (the last runs are rebased onto older commits)
  • Issue pending a real fix: Investigate flaky addons/stringbytes-external-exceed-max/test-stringbytes-external-exceed-max-by-1-binary on FreeBSD 11 node#22317
• async-hooks/test-callback-error and async-hooks/test-statwatcher
  • Marked as flaky in test: mark flaky tests as flaky node#22330
• parallel/test-crypto-scrypt and parallel/test-tls-passphrase
  • All on containers with shared OpenSSL, may be fixed by ansible: upgrade to OpenSSL 1.1.0i & 1.0.2p in sharedlibs build#1451
  • Fix in Upgrade to OpenSSL-1.1.0i node#22318
• parallel/test-gc-http-client
  • Stopped appearing in recent runs, maybe wait for a bit more before looking into it It appeared in the daily master CI. Issue Investigate flaky parallel/test-gc-http-client node#22336
• Changes not staged for commit
  • Discussed in this week's build WG meeting: WIP: doc: add minutes August 14 build#1448
  • Should be fixed once we set config.autocrlf = false on the workers
  • Issue link: Modified files with CRLF block CI build#1443
  • Discussion in Setting autocrlf and safecrlf on workers build#1453
  • Another proposed fix: deps: fix CRLF in text file not present in upstream node#22340
• fatal: Couldn't find remote ref refs/heads/v4.x-staging
  • All happened on the PIs, looks like some of the tests were not being run @nodejs/build

[joyeecheung]

Move to #14