test-requireio-osx1010-x64-1 is borked

@rvagg

I took it offline for now https://ci.nodejs.org/computer/test-requireio-osx1010-x64-1/

It started with perma:

Details

FATAL: command execution failed
java.nio.channels.ClosedChannelException
	at org.jenkinsci.remoting.protocol.NetworkLayer.onRecvClosed(NetworkLayer.java:154)
	at org.jenkinsci.remoting.protocol.impl.NIONetworkLayer.ready(NIONetworkLayer.java:179)
	at org.jenkinsci.remoting.protocol.IOHub$OnReady.run(IOHub.java:721)
	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused: hudson.remoting.ChannelClosedException: channel is already closed
	at hudson.remoting.Channel.send(Channel.java:605)
	at hudson.remoting.Request.call(Request.java:130)
	at hudson.remoting.Channel.call(Channel.java:829)
	at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:257)
	at com.sun.proxy.$Proxy59.isAlive(Unknown Source)
	at hudson.Launcher$RemoteLauncher$ProcImpl.isAlive(Launcher.java:1043)
	at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:1035)
	at hudson.tasks.CommandInterpreter.join(CommandInterpreter.java:155)
	at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:109)
	at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:66)
	at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
	at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:779)
	at hudson.model.Build$BuildExecution.build(Build.java:206)
	at hudson.model.Build$BuildExecution.doRun(Build.java:163)
	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:534)
	at hudson.model.Run.execute(Run.java:1728)
	at hudson.matrix.MatrixRun.run(MatrixRun.java:146)
	at hudson.model.ResourceController.execute(ResourceController.java:98)
	at hudson.model.Executor.run(Executor.java:405)
Build step 'Execute shell' marked build as failure

I logged in and made the mistake of rimrafing the workspace. The job setup is broken as it refers to /home/iojs/io.js.reference when the local mirror is actually at /Users/iojs/git/io.js.reference.
So I cloned the local mirror manually and got one step further to.

Details

 > git config remote.origin.url https://github.com/nodejs/node.git # timeout=10
Fetching upstream changes from https://github.com/nodejs/node.git
 > git --version # timeout=10
using GIT_SSH to set credentials 
 > git fetch --tags --progress https://github.com/nodejs/node.git +refs/heads/*:refs/remotes/origin/* +refs/pull/15083/head:refs/remotes/origin/_jenkins_local_branch # timeout=20
Checking out Revision aaec74c55f00a233206059dc3cd69242e2e3a643 (refs/remotes/origin/_jenkins_local_branch)
java.lang.NoClassDefFoundError: Could not initialize class jenkins.model.Jenkins$MasterComputer
	at org.jenkinsci.plugins.gitclient.AbstractGitAPIImpl.withRepository(AbstractGitAPIImpl.java:29)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.withRepository(CliGitAPIImpl.java:71)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:895)
	at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:870)
	at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:829)
	at hudson.remoting.UserRequest.perform(UserRequest.java:153)
	at hudson.remoting.UserRequest.perform(UserRequest.java:50)
	at hudson.remoting.Request$2.run(Request.java:336)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at hudson.remoting.Engine$1$1.run(Engine.java:94)
	at java.lang.Thread.run(Thread.java:745)
	at ......remote call to JNLP4-connect connection from static-n49-176-213-208.sbr2.nsw.optusnet.com.au/49.176.213.208:52267(Native Method)
	at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1545)
	at hudson.remoting.UserResponse.retrieve(UserRequest.java:253)
	at hudson.remoting.Channel.call(Channel.java:830)
Caused: java.io.IOException: Remote call on JNLP4-connect connection from static-n49-176-213-208.sbr2.nsw.optusnet.com.au/49.176.213.208:52267 failed
	at hudson.remoting.Channel.call(Channel.java:838)
	at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:257)
	at com.sun.proxy.$Proxy60.withRepository(Unknown Source)
	at org.jenkinsci.plugins.gitclient.RemoteGitImpl.withRepository(RemoteGitImpl.java:235)
	at hudson.plugins.git.GitSCM.printCommitMessageToLog(GitSCM.java:1195)
	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1159)
	at hudson.scm.SCM.checkout(SCM.java:496)
	at hudson.model.AbstractProject.checkout(AbstractProject.java:1281)
	at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:604)
	at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:529)
	at hudson.model.Run.execute(Run.java:1728)
	at hudson.matrix.MatrixRun.run(MatrixRun.java:146)
	at hudson.model.ResourceController.execute(ResourceController.java:98)
	at hudson.model.Executor.run(Executor.java:405)
Run condition [Always] enabling perform for step [[]]

But there I'm stuck...

/cc @rvagg @joaocgreis

fwiw the lack of a git mirror isn't essential and should just be a quiet fail, while it'll slow it down if it's not in the right place it still should be fine.

I'm still messing with this machine, some weird network problems, can't even update the git mirror or clone a new one using --mirror or --bare although I have no idea if that's related! I've tried updating the machine, rebooting, rebooting jenkins master, deleting and recreating the node config on jenkins master, rebooting network hardware the machine is plugged in to. All to no avail.

I would try updating Java but a recent VMWare Fusion upgrade on that machine messed up access to the consoles, I only have black screens and have to use SSH to do everything like everyone else.

I'm out for 12+ hours but will look back at this soon. I'm not going to leave this machine offline because failures are better than a big queue in jenkins. If anyone else wants to tinker on the machine please do so.

@nodejs/build did anyone install a new jenkins plugin recently?

ok, really heading off now, my remaining theory is that a plugin was installed recently that's inteferring. If someone installed one in the last week or so, I'd appreciate you either drop a note in here or experiment with removing it and running node-test-commit-osx.

The fact that there are only two updates available suggests that the plugins were updated recently, so it might have been an update that changed things.

I haven't installed anything since the rebuild plugin (which was apparently in July #672).

The node-test-commit-osx jobs seem to be trying to build on test-packetnet-ubuntu1604-x64-1 which seems to minimally be a labeling error, but maybe a full-on larger problem? See https://ci.nodejs.org/job/node-test-commit-osx/12132/ for an example.

@Trott

@Trott is that a problem? The parent job is running there because it has the jenkins-workspace, but that just runs the initial git clone. The actual job is running on test-requireio-osx1010-x64-1.

@Trott

@Trott is that a problem? The parent job is running there because it has the jenkins-workspace, but that just runs the initial git clone. The actual job is running on test-requireio-osx1010-x64-1.

Nope, not a problem if it works. :-D

@gibfahn

@gibfahn I updated the plugins while trying to sort this out, which is why they look like that. Didn't help.

Interestingly there have been a few greens today and the last run only disconnected as it was already into the test running.

I'll see if I can figure out how to determine which plugins were most recently installed and start removing them to see if that makes a difference. Another alternative might be to try and get a different JVM installed but as I've already mentioned that's going to be tricky without a GUI!

some more weird https errors on my local infra, this time on pi1's building binaries: https://ci-release.nodejs.org/job/iojs+release/1990/nodes=pi1-raspbian-wheezy/console (only accessible to people with release access)

hudson.plugins.git.GitException: Failed to fetch from https://github.com/nodejs/node.git

I haven't experienced anything similar on my own use of this network (I'm on it now) but this is very similar to what I was experiencing on the macOS box on the same network! I've done a cleanup on those two release machines and will keep an eye on those builds, I could conceive of a way this might be local to those machines but it's pretty suspicious on the face of it.

Re macOS, I've downloaded the latest Oracle JRE macOS tarball and unpacked it into /usr/local/lib with a /usr/local/bin/java symlink and changed the slave.jar startup to use that binary and it's running now. It's going @ https://ci.nodejs.org/job/node-test-commit-osx/12159/nodes=osx1010/console but taking a very long time to even get started which makes me still suspect a problematic plugin.

Will keep an eye on all of this and continue to work out what on earth is up. Again, any help or suggestions would be appreciated!

@rvagg

@rvagg is there anything in the master jenkins logs? If it's to do with the git clone it might be an issue moving clones around. Not sure what else could be going on here, haven't been seeing this on our internal Jenkins instance at all.

Hm, yeah, a bunch of these just now for that host only java.util.concurrent.TimeoutException when trying to do the regular monitoring.

Maybe I just need to and shut everything down, including the internet connection and start it all up again slowly.

Another alternative might be to try and get a different JVM installed but as I've already mentioned that's going to be tricky without a GUI!

RE GUI, did you try to VNC to the VM? I have some success with that...

@refack

@refack got some command-line-fu to enable screen sharing? it's not enabled on the VMs unfortunately.

@joaocgreis

@joaocgreis can you look at the shared binary repo and see if you can make it prune any more aggressively? It's at 3.2G at the moment and may be contributing somewhat to network problems (not likely the cause of this particular problem though).

@rvagg

An old favorite is raising it's head:

not ok 75 async-hooks/test-callback-error
  ---
  duration_ms: 15.545
  severity: fail
  stack: |-
    start case 1
    end case 1: 109.402ms
    start case 2
    end case 2: 99.464ms
    start case 3
    end case 3: 15.766ms
    Error: test_callback_abort
        at ActivityCollector.initHooks.oninit.common.mustCall (/Users/iojs/build/workspace/node-test-commit-osx/nodes/osx1010/test/async-hooks/test-callback-error.js:36:45)
        at ActivityCollector.oninit (/Users/iojs/build/workspace/node-test-commit-osx/nodes/osx1010/test/common/index.js:509:15)
        at ActivityCollector._init (/Users/iojs/build/workspace/node-test-commit-osx/nodes/osx1010/test/async-hooks/init-hooks.js:182:10)
        at emitInitNative (async_hooks.js:446:43)
        at Object.emitInitScript [as emitInit] (async_hooks.js:349:3)
        at Object.<anonymous> (/Users/iojs/build/workspace/node-test-commit-osx/nodes/osx1010/test/async-hooks/test-callback-error.js:38:17)
        at Module._compile (module.js:549:30)
        at Object.Module._extensions..js (module.js:560:10)
        at Module.load (module.js:483:32)
        at tryModuleLoad (module.js:446:12)
     1: node::Abort() [/Users/iojs/build/workspace/node-test-commit-osx/nodes/osx1010/out/Release/node]
     2: node::Chdir(v8::FunctionCallbackInfo<v8::Value> const&) [/Users/iojs/build/workspace/node-test-commit-osx/nodes/osx1010/out/Release/node]
     3: v8::internal::FunctionCallbackArguments::Call(void (*)(v8::FunctionCallbackInfo<v8::Value> const&)) [/Users/iojs/build/workspace/node-test-commit-osx/nodes/osx1010/out/Release/node]
     4: v8::internal::MaybeHandle<v8::internal::Object> v8::internal::(anonymous namespace)::HandleApiCallHelper<false>(v8::internal::Isolate*, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::FunctionTemplateInfo>, v8::internal::Handle<v8::internal::Object>, v8::internal::BuiltinArguments) [/Users/iojs/build/workspace/node-test-commit-osx/nodes/osx1010/out/Release/node]
     5: v8::internal::Builtin_Impl_HandleApiCall(v8::internal::BuiltinArguments, v8::internal::Isolate*) [/Users/iojs/build/workspace/node-test-commit-osx/nodes/osx1010/out/Release/node]
     6: 0x16a6e9e846fd

Need to disable the crash reporter nodejs/node#13527 (comment)

This has been fixed since @rvagg ran these two commands on the host:
launchctl unload -w /System/Library/LaunchAgents/com.apple.ReportCrash.plist
sudo launchctl unload -w /System/Library/LaunchDaemons/com.apple.ReportCrash.Root.plist
Note that they need to be run from a GUI login or else they error out.

There's an issue open to get this into the provisioning for OS X machines. But for now, the problem is resolved, so I'm going to close this.

P.S. it's rejecting the test key:

debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,keyboard-interactive
debug1: Next authentication method: publickey
debug1: Trying private key: ▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄
debug2: we sent a publickey packet, wait for reply
debug1: Authentications that can continue: publickey,keyboard-interactive
debug2: we did not send a packet, disable method
debug1: Next authentication method: keyboard-interactive
debug2: userauth_kbdint
debug2: we sent a keyboard-interactive packet, wait for reply
debug2: input_userauth_info_req
debug2: input_userauth_info_req: num_prompts 1
Password:
Killed by signal 2.

@rvagg

@rvagg I cleaned the binary repo (can only do manually, because there's no way to tell if a branch is in use or not). Many branches were left behind when jobs were aborted, so I took a look and was able to change the way branches are deleted in arm-fanned and windows-fanned to run even if the job is aborted.

@refack

@refack thanks, just ran those commands via ssh (no security complaints, didn't need to do it via UI!), this re-run of a previously failed job (due to that error) has an unrelated failure: https://ci.nodejs.org/job/node-test-commit-osx/12210/nodes=osx1010/console, so I think we might be good now. Will watch today's daily jobs to see if we get more green. I've updated my scripted list above to include these. 🤞

1 green job https://ci.nodejs.org/job/node-test-commit-osx/12211/
Which IMO is a very possitive sign 🏆

2nd green job in a row https://ci.nodejs.org/job/node-test-commit-osx/12212/
By Jove I think she got it

P.S. Let's get those commands added to ansible or whatever, yeah?

👍

Something's still up. The 3 failures in that list of most recent builds from today are disconnects during compile. Not super-early like before but still the same kind of disconnect.

Still mostly green but regular disconnects it seems. I haven't looked any further back at the failures other than today's, I only just caught the machine offline so went for a quick dig. If anyone else wants to, go to https://ci.nodejs.org/computer/test-requireio-osx1010-x64-1/builds, wait for it to load and click through to each of them and look at the console output to see if it's a test failure or disconnection.

I don't have an answer here except that we need to get on with our mac IaaS work and have more of these for redundancy and hopefully stability.

@rvagg

@rvagg @joaocgreis
test-requireio-osx1010-x64-1 is still rejecting the test key for iojs

I can't connect either. Missing nodejs_build_test?

yes, sorry, new rebuild missed out on key, fixed now, reopen if it doesn't work for y'all

Navigation Menu

Repository navigation

test-requireio-osx1010-x64-1 is borked #861

6 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

test-requireio-osx1010-x64-1 is borked #861

Description

Activity

rvagg commented on Sep 2, 2017

rvagg commented on Sep 2, 2017

rvagg commented on Sep 2, 2017

gibfahn commented on Sep 2, 2017

Trott commented on Sep 2, 2017

gibfahn commented on Sep 2, 2017

Trott commented on Sep 3, 2017

rvagg commented on Sep 4, 2017

rvagg commented on Sep 5, 2017

gibfahn commented on Sep 5, 2017

rvagg commented on Sep 5, 2017

refack commented on Sep 5, 2017

rvagg commented on Sep 5, 2017

rvagg commented on Sep 5, 2017

6 remaining items

refack commented on Sep 5, 2017

refack commented on Sep 5, 2017

joaocgreis commented on Sep 5, 2017

rvagg commented on Sep 7, 2017

refack commented on Sep 7, 2017

refack commented on Sep 7, 2017

Trott commented on Sep 7, 2017

rvagg commented on Sep 7, 2017

rvagg commented on Sep 11, 2017

refack commented on Oct 2, 2017

joaocgreis commented on Oct 3, 2017

rvagg commented on Oct 23, 2017

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions