Investigate parallel/test-cluster-send-handle-large-payload failure on FreeBSD #14844

Activity

1. Trott

   added

   flaky-testIssues and PRs involving tests that fail intermittently in CI.Issues and PRs involving tests that fail intermittently in CI.

   clusterIssues and PRs related to the cluster subsystem.Issues and PRs related to the cluster subsystem.

   freebsdIssues and PRs related to the FreeBSD platform.Issues and PRs related to the FreeBSD platform.

   on Aug 15, 2017
2. addaleax

   self-assigned this

   on Aug 15, 2017
3. addaleax commented on Aug 15, 2017

   

   addaleax

   on Aug 15, 2017

   Member

   More actions

   sigh

   Not sure how this can happen, but at least this is an OS I can get access to.
4. addaleax commented on Aug 15, 2017

   

   addaleax

   on Aug 15, 2017

   Member

   More actions

   Stress test to get an idea of just how flaky this is: https://ci.nodejs.org/job/node-stress-single-test/1375/nodes=freebsd10-64/
5. BridgeAR commented on Sep 28, 2017

   

   BridgeAR

   on Sep 28, 2017

   Member

   More actions

   It happened again recently.
6. addaleax

   removed their assignment

   on Dec 8, 2017
7. MylesBorins commented on Feb 22, 2018

   

   MylesBorins

   on Feb 22, 2018

   Contributor

   More actions

   happened again https://ci.nodejs.org/job/node-test-commit-freebsd/15723/nodes=freebsd11-x64/consoleFull
8. Trott commented on Mar 12, 2018

   

   Trott

   on Mar 12, 2018

   MemberAuthor

   More actions

   Happened just now: https://ci.nodejs.org/job/node-test-commit-freebsd/16120/nodes=freebsd11-x64/console
9. Trott commented on Mar 12, 2018

   

   Trott

   on Mar 12, 2018

   MemberAuthor

   More actions

   Stress test, once in sequence and once with -J for parallelism.

   Sequence: https://ci.nodejs.org/job/node-stress-single-test/1787/nodes=freebsd11-x64/
   Parallel: https://ci.nodejs.org/job/node-stress-single-test/1788/nodes=freebsd11-x64/
10. Trott commented on Mar 12, 2018

    

    Trott

    on Mar 12, 2018

    MemberAuthor

    More actions

    I messed up the parallel stress test. Here it is again, hopefully correctly done this time:

    https://ci.nodejs.org/job/node-stress-single-test/1789/nodes=freebsd11-x64/
11. Trott commented on Mar 13, 2018

    

    Trott

    on Mar 13, 2018

    MemberAuthor

    More actions

    Parallel results suggest this is an issue about resource constraints and it might be solved (or mitigated) by moving the test to sequential. I'll submit a PR to do that.

    ok 1 parallel/test-cluster-send-handle-large-payload
      ---
      duration_ms: 2.853
      ...
    ok 2 parallel/test-cluster-send-handle-large-payload
      ---
      duration_ms: 3.294
      ...
    ok 3 parallel/test-cluster-send-handle-large-payload
      ---
      duration_ms: 3.316
      ...
    ok 4 parallel/test-cluster-send-handle-large-payload
      ---
      duration_ms: 3.172
      ...
    not ok 5 parallel/test-cluster-send-handle-large-payload
      ---
      duration_ms: 3.92
      severity: fail
      stack: |-
        undefined:1
        aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
        
        SyntaxError: Unexpected token a in JSON at position 0
            at JSON.parse (<anonymous>)
            at Pipe.channel.onread (internal/child_process.js:490:28)
      ...
    ok 6 parallel/test-cluster-send-handle-large-payload
      ---
      duration_ms: 3.236
12. Trott

    added a commit that references this issue on Mar 13, 2018

    test: fix test-cluster-send-handle-large-payload

    ...

    2403bfb
13. Trott commented on Mar 13, 2018

    

    Trott

    on Mar 13, 2018

    MemberAuthor

    More actions

    (Sequential stress test turned up green. It was just the parallel stress test that showed the failures.)
14. santigimeno commented on Mar 13, 2018

    

    santigimeno

    on Mar 13, 2018

    Member

    More actions

    I looked into this a while ago and IIRC it seemed to me that it could be related with a bug in the freeBSD kernel where data was silently lost when sending SCM_RIGHTS messages with large payload. See: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=181741 and https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=215933.

    👍React with 👍1Reacted by Rich Trott
15. Trott

    added a commit that references this issue on Mar 15, 2018

    test: fix test-cluster-send-handle-large-payload

    ...

    5e23b65
16. Trott

    closed this as completedin 5e23b65on Mar 15, 2018
17. targos

    added a commit that references this issue on Mar 17, 2018

    test: fix test-cluster-send-handle-large-payload

    ...

    03feca3
18. MylesBorins

    added a commit that references this issue on Mar 20, 2018

    test: fix test-cluster-send-handle-large-payload

    ...

    9aa5090
19. MylesBorins

    added

    backported-to-v8.x

    on Jul 31, 2018
20. MylesBorins

    added a commit that references this issue on Jul 31, 2018

    test: fix test-cluster-send-handle-large-payload

    ...

    093b777
21. rvagg

    added a commit that references this issue on Aug 16, 2018

    test: fix test-cluster-send-handle-large-payload

    ...

    0517cd8