Compilation broken on Windows in the CI

node-compile-windows and node-compile-windows-debug have been constantly failing since the weekend began. I'm currently investigating it and the reason seems to be the latest update of VS 2022 to 17.10.0 (everything was fine with 17.9.7). While I'm working on it, those jobs/machines will be down for some time, so just letting everyone know in advance. I'll write update here once I have something.

I tried to compile locally out of curiosity. This is what I get:

  maglev.cc
C:\Users\mzasso\git\node\deps\v8\src\maglev\maglev-ir.cc(3855): fatal  error C1001: Internal compiler error. [C:\Users\
mzasso\git\node\tools\v8_gypfiles\v8_base_without_compiler.vcxproj]
  (compiler file 'D:\a\_work\1\s\src\vctools\Compiler\Utc\src\p2\main.c', line 242)
   To work around this problem, try simplifying or changing the program near the locations listed above.
  If possible please provide a repro here: https://developercommunity.visualstudio.com
  Please choose the Technical Support command on the Visual C++
   Help menu, or open the Technical Support help file for more information
    CL!RaiseException()+0x54
    CL!DllGetObjHandler()+0x31a0c
    CL!DllGetObjHandler()+0x448
    CL!DllGetObjHandler()+0x5c2690
    CL!DllGetObjHandler()+0x6cf368
    CL!DllGetObjHandler()+0x7c7a74
    CL!DllGetObjHandler()+0x7bce74
    CL!DllGetObjHandler()+0x6cd960
    CL!DllGetObjHandler()+0x394fbc
    CL!DllGetObjHandler()+0x114b0
    CL!DllGetObjHandler()+0x85d0
    CL!DllGetObjHandler()+0x18c88
    CL!DllGetObjHandler()+0x3fb918
    CL!strtol()+0xac
    CL!BaseThreadInitThunk()+0x30
    CL!RtlUserThreadStart()+0x3c

I've managed to fix the compilation jobs. The problem was as I suspected the update to VS 17.10.0. Luckily, the VS community edition that we use offers rollback, so I took it back to 17.9.7 and pinned that version. There is also another possibility: to install VS2019 build tools with the exact version and then use VS only for the editor and add workloads, components, etc. from the build tools installation.

vcbuild.bat and vswhere_usability_wrapper.cmd are written so they only accept the latest version of build tools, so if you have VS 17.10.0, Node will not let you compile with eg v17.9.7 compiler. Since ClangCL is coming (hopefully) soon, I do not think this is worth fixing to support various versions of MSVC, but I'll keep it on my radar just in case.

The new problem I see now is that node-test-binary-windows-native-suites are constantly failing since yesterday (they were not running 1-2 days before that because of the compilation issue), so I'll dig into that.

The fix was applied yesterday and everything seems to be back to normal, Some PRs will need to be rebased to pass the CI, but other than that, everything else is working normally.

I will keep this issue open for a few more days, in case something goes wrong. Planning to close it on Friday.

What's the long term solution ? Is Microsoft aware of the MSVC bug?

Broken all over the place... The only "solution" with the current MSVC release is to #pragma optimize off/on around the problem function. REALLY not good-- this is impact MANY projects (including Unreal Engine)

@targos

Hi @targos @StefanStojanovic:
I work for the MSVC optimizer team, and we want to look into the above failure and fix it.

We'd really appreciate if you can provide a repro of the failure (ex: a pre-processed Cpp file and cl command line used) so that we can investigate the compiler error. Please file an issue with the repro at: https://developercommunity.visualstudio.com/cpp/report. Thanks.
This will help to resolve the issue and unblock the VS-toolset update.

Thanks.

I pasted how to repro.. it’s a public repository “Vahalla Game” on but.. I just got an msvc update about 2 hrs ago, and it’s working properly again, so looks like someone fixed it… but double check me on that. The issue was in “RenderSky.cpp” in the draw() function. Mind you this isn’t my code, I use it for a demo/unit test for compatibility. Happy to provide more info if you need it. Sent from [Proton Mail](https://proton.me/mail/home) for iOS

…

On Wed, May 29, 2024 at 4:16 PM, Swaroop Sridhar ***@***.***(mailto:On Wed, May 29, 2024 at 4:16 PM, Swaroop Sridhar <<a href=)> wrote: Hi ***@***.***(https://github.com/targos) ***@***.***(https://github.com/StefanStojanovic): I work for the MSVC optimizer team, and we want to look into the above failure and fix it. We'd really appreciate if you can provide a repro of the failure (ex: a pre-processed Cpp file and cl command line used) so that we can investigate the compiler error. Please file an issue with the repro at: https://developercommunity.visualstudio.com/cpp/report. Thanks. This will help to resolve the issue and unblock the VS-toolset update. Thanks. — Reply to this email directly, [view it on GitHub](#3739 (comment)), or [unsubscribe](https://github.com/notifications/unsubscribe-auth/BEZUO2RFKSLJOWKJQML4QTLZEZATPAVCNFSM6AAAAABIJUKI2SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMZYGI3TOOBWGY). You are receiving this because you commented.Message ID: ***@***.***>

What's the long term solution ? Is Microsoft aware of the MSVC bug?

I've brought this up in the conversation with my contact at Microsoft. He told me, he'll look into it and get back to me. I've also noticed there is a new version of VS v17.10.1 (still not in Chocolatey, but will be there in a few days probably). I'll try it locally to see if it compiles correctly.

@StefanStojanovic

Hi @StefanStojanovic it is still useful if you can file a developer community ticket with the repro at: https://developercommunity.visualstudio.com/cpp/report
This way, the issue will be officially tracked and resolved.

Thanks.

I tried the new version and it fails as well. I'll need some time to find a way to generate a preprocessed file to submit to the community forums

I don't really know what to do. As soon as I add /P to the MSVC arguments, compilation fails saying that it cannot find v8_pch.h.

There's a 2nd problem with these fixes [Compiler Version Mismatch:: There's a side-effect to the last "Fix" that sort-of fixed the optimization having an internal error-- Now, libraries built won't link on older (as in 1 week older) VS 2022, meaning you will need to deploy LIBs (if you deploy libs to customer) for 2022 prior to Mid-May 2022, AND Libs AFTER Mid-May 2022. The unit-test is quite easy.. just build any project with MSVC Prior to the last 2 updates, and create and link to a LIB made in the latest MSVC, and you'll get an error saying the LIB and OBJs have a compiler-version mismatch. Forcing customers to "upgrade" their existing compiler is not acceptable either (production machines simply won't do that on a whim). Microsoft Visual Studio Professional 2022 Version 17.10.1 VisualStudio.17.Release/17.10.1+34928.147 Microsoft .NET Framework Version 4.8.09032 Installed Version: Professional Visual C++ 2022 00483-10050-07952-AA543 Microsoft Visual C++ 2022 ASP.NET and Web Tools 17.10.338.1105 ASP.NET and Web Tools Azure App Service Tools v3.0.0 17.10.338.1105 Azure App Service Tools v3.0.0 C# Tools 4.10.0-3.24270.2+e8f775c1d8a73dee7ad02408712d714251e708ea C# components used in the IDE. Depending on your project type and settings, a different version of the compiler may be used. Extensibility Message Bus 1.4.39 ***@***.***) Provides common messaging-based MEF services for loosely coupled Visual Studio extension components communication and integration. GitHub Copilot 0.2.889.30432 GitHub Copilot is an AI pair programmer that helps you write code faster and with less work. Linux Core Dump Debugging 1.0.9.34902 Enables debugging of Linux core dumps. Microsoft JVM Debugger 1.0 Provides support for connecting the Visual Studio debugger to JDWP compatible Java Virtual Machines NuGet Package Manager 6.10.0 NuGet Package Manager in Visual Studio. For more information about NuGet, visit https://docs.nuget.org/ Razor (ASP.NET Core) 17.10.3.2427201+4f57d1de251e654812adde201c0265a8ca7ca31d Provides languages services for ASP.NET Core Razor. Test Adapter for Boost.Test 1.0 Enables Visual Studio's testing tools with unit tests written for Boost.Test. The use terms and Third Party Notices are available in the extension installation directory. Test Adapter for Google Test 1.0 Enables Visual Studio's testing tools with unit tests written for Google Test. The use terms and Third Party Notices are available in the extension installation directory. TypeScript Tools 17.0.30327.2001 TypeScript Tools for Microsoft Visual Studio Visual Basic Tools 4.10.0-3.24270.2+e8f775c1d8a73dee7ad02408712d714251e708ea Visual Basic components used in the IDE. Depending on your project type and settings, a different version of the compiler may be used. Visual C++ for Linux Development 1.0.9.34902 Visual C++ for Linux Development Visual F# Tools 17.10.0-beta.24228.1+dd749058c91585e9b5dae62b0f8df892429ee28f Microsoft Visual F# Tools Visual Studio IntelliCode 2.2 AI-assisted development for Visual Studio. VisualStudio.DeviceLog 1.0 Information about my package VisualStudio.Mac 1.0 Mac Extension for Visual Studio VSPackage Extension 1.0 VSPackage Visual Studio Extension Detailed Info Xamarin 17.10.0.110 ***@***.***) Visual Studio extension to enable development for Xamarin.iOS and Xamarin.Android. Xamarin Designer 17.10.3.10 ***@***.***) Visual Studio extension to enable Xamarin Designer tools in Visual Studio. Xamarin.Android SDK 13.2.2.0 (d17-5/45b0e14) Xamarin.Android Reference Assemblies and MSBuild support. Mono: d9a6e87 Java.Interop: ***@***.*** SQLite: ***@***.*** Xamarin.Android Tools: ***@***.*** Sent with [Proton Mail](https://proton.me/) secure email.

…

On Friday, May 31st, 2024 at 3:57 AM, Michaël Zasso ***@***.***> wrote: I don't really know what to do. As soon as I add /P to the MSVC arguments, compilation fails saying that it cannot find v8_pch.h. — Reply to this email directly, [view it on GitHub](#3739 (comment)), or [unsubscribe](https://github.com/notifications/unsubscribe-auth/BEZUO2VIJXBTDBAAARQKQMDZFA3QHAVCNFSM6AAAAABIJUKI2SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNBRGUZTQOBTGE). You are receiving this because you commented.Message ID: ***@***.***>

doc: warn for windows build bug

Good news: Visual Studio 17.11.0 was just released and the build works again (at least on my PC)!

Good news: Visual Studio 17.11.0 was just released and the build works again (at least on my PC)!

I tested it as well and it works for me too. We can wait a week or two, for safety and then unblock VS upgrade in the CI and eventually reenable GH action for building Windows.

For the vision of Visual Studio used in windows-latest/window-2022 environment, you can check it on https://github.com/actions/runner-images/blob/main/images/windows/Windows2022-Readme.md#visual-studio-enterprise-2022.

And from the commit history, the update frequence seems to be a week.

For the vision of Visual Studio used in windows-latest/window-2022 environment, you can check it on https://github.com/actions/runner-images/blob/main/images/windows/Windows2022-Readme.md#visual-studio-enterprise-2022.

actions/runner-images#10448

It was deployed on GitHub Actions: actions/runner-images#10448 (comment)

Here's a run on my fork: https://github.com/targos/node/actions/runs/10643236538

It passed so I enabled https://github.com/nodejs/node/actions/workflows/build-windows.yml and https://github.com/nodejs/node/actions/workflows/coverage-windows.yml.

@StefanStojanovic

GH Actions runs are green. @StefanStojanovic I let you close this as soon as Jenkins hosts are updated.

@targos

Thanks for enabling that @targos that's great! However, updating Jenkins CI won't be as straightforward as removing the Chocoaltey pin for VS and upgrading. While working on nodejs/node#54655 I did a lot of builds for all of the combinations (MSVC/ClangCL + x64/ARM64). I noticed that VS 17.11 MSVC + ARM64 doesn't compile (on VS 17.9 all works fine). The error I get is this:

E:\work\node\deps\ngtcp2\nghttp3\lib\nghttp3_ringbuf.c(38,14): error C2169: '__popcnt': intrinsic function, cannot be defined [E:\work\node\deps\ngtcp2\nghttp3.vcxproj]
E:\work\node\deps\ngtcp2\ngtcp2\lib\ngtcp2_ringbuf.c(36,21): error C2169: '__popcnt': intrinsic function, cannot be defined [E:\work\node\deps\ngtcp2\ngtcp2.vcxproj]

I'm OOF this week, so when I'm back I'll open an issue for this, but we'll probably need to submit a fix upstream since this problem is with the dependency.

doc: warn for windows build bug

This was handled a long time ago, just not closed. Closing it now. Also there was another blocker (v17.12) which has its own issue.

Navigation Menu

Repository navigation

Compilation broken on Windows in the CI #3739

42 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Compilation broken on Windows in the CI #3739

Description

Activity

targos commented on May 26, 2024

StefanStojanovic commented on May 27, 2024

StefanStojanovic commented on May 28, 2024

targos commented on May 28, 2024

VXACDev commented on May 28, 2024

swaroop-sridhar commented on May 29, 2024

VXACDev commented on May 30, 2024

StefanStojanovic commented on May 30, 2024

swaroop-sridhar commented on May 30, 2024

targos commented on May 30, 2024

targos commented on May 31, 2024

VXACDev commented on May 31, 2024

42 remaining items

targos commented on Aug 15, 2024

StefanStojanovic commented on Aug 16, 2024

liudonghua123 commented on Aug 17, 2024

richardlau commented on Aug 17, 2024

targos commented on Aug 31, 2024

targos commented on Aug 31, 2024

targos commented on Aug 31, 2024

StefanStojanovic commented on Sep 2, 2024

StefanStojanovic commented on Dec 2, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions