I thought about writing some guides on this topic in the nodejs/node repo, but it’s easier to throw whatever tricks I personally use on the Internet first - I am also going to heavily use the pronouns “I”, “We” and “You” in this post, and talk about my personal preference here, both of which we always try to avoid in the repo ;)

Note that many hidden features mentioned in this post are not meant to be used in the user land of Node.js, and there is zero stability guarantee about them. Do not rely on them in any project outside the scope of Node.js core.

An introduction to Node.js core development workflow

You can skip this section if you already know about the basic structure of the Node.js source code and how to build it.

The source code of Node.js itself is primarily written in C++ and JavaScript. The JS part is mostly under the lib directory and the C++ part is mostly under the src directory. There are also a bunch of third-party dependencies like v8 or libuv which are placed under the deps directory. This post covers the techniques on how to debug most of them. There are also things like Python scripts and DTrace scripts in the code base but I don’t usually need to (or, I don’t really know how to) debug them with anything fancy, so I won’t cover those in this post.

The Node.js release binaries do not strip the debug symbols (surprised?) so you don’t necessarily have to build a debug build to get meaningful C/C++ stack traces or to do step debugging in a native debugger like LLDB or GDB. That said the release binary is built with -Os so not everything is available in the release build, and you do need to use a debug build if you want to use V8’s internal debug utilities.

The debug build can be configured by adding --debug when you run the configure script before building Node.js (on non-Windows systems at least, the vcbuild.bat script we use on Windows runs configure for you), and will be built as out/Debug/node (symlinked as ./node_g in the project directory). So typically this is what I do:

1
2
3
4
5
6
7
8
9
10
11
# For release builds
cd /path/to/node/project
./configure
make -j8
./node test.js  # or out/Release/node test.js

# For debug builds:
cd /path/to/node/project
./configure --debug
make -j8
./node_g test.js  # or out/Debug/node test.js

For more information check out BUILDING.md in the project directory.

Tips on using VSCode to develop Node.js core

The C++ intellisense offered by the cpptools plugin can consume a compilation command database for better understanding of the code base and the build flags that you are using. To generate the database for Node.js, add a -C switch when you run configure. For example, to generate the database for the release build, run

1
./configure -C

in the project directory, and in .vscode/c_cpp_properties.json, add

1
"compileCommands": "${workspaceFolder}/out/Release/compile_commands.json"

to your configurations. Or do ./configure --debug -C and replace Release with Debug in the snippet above if you are using debug builds.

On how to integrate the plugin itself in VSCode, see the docs of the plugin for details.

Debugging in JS: V8 Inspector

This is what I tend to use to debug in the JS land these days because it works with most Node.js binaries I need to debug.

Usually I would write a test script in a local file, say, test.js, to trigger the code I want to execute, and run

1
./node --inspect-brk test.js

Then I usually open up Chrome, type chrome://inspect in the address bar, and use the Node.js dedicated DevTools UI, where the execution should stop at the first line of test.js - it is possible to use other clients like Visual Studio Code or node inspect that is bundled into the binary, but my personal reference is to use Chrome to debug JS and use VSCode to debugger C++ (more on that later). Note that with certain combinations of Node.js and Chrome certain features may be broken in the inspector, and sometimes you’ll need to use the Chrome Canary to get that fix early before it enters the stable channel of Chrome (example).

Sometimes when I need to debug into the bootstrap process of Node.js, I use the hidden switch introduced by danbev for Node.js core developers:

1
./node --inspect-brk-node test.js

Which is supposed to break into the first line of the bootstrapping JS code of Node.js, currently this is implemented as an explicit debugger; statement in lib/internal/bootstrap/loaders.js. Theoretically that’s not the first JS script we execute during bootstrap (that would be lib/internal/per_context.js at the time this post was written), but it’s the first JS script that’s executed after we bootstrap to a point where the inspector can be started.

If you want to set a breakpoint programmatically, adding the debugger; statement in the JS code would do the trick. Note that you need to recompile the binary to see the changes because the builtin JS code in Node.js are compiled into the binary as static C char arrays instead of being loaded from disk. Also, remember to remove these statements before you commit and send a PR.

Native Debuggers for C/C++

There are only two debuggers I personally use: gdb (on Linux) and lldb (on macOS and sometimes Linux). I don’t usually develop Node.js core on Windows but the debugger in Visual Studio should also work just fine, though I am not aware of any special tricks that can be used in VS.

I typically use the vscode-lldb plugin of Visual Studio Code for daily development. This is what my launch.json usually looks like

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
{
  "version": "0.2.0",
  "configurations": [
    {
      "name": "(lldb) Launch",
      "type": "lldb",
      "request": "launch",
      "program": "${workspaceFolder}/out/Release/node",
      "args": [
        "--expose-internals",
        "test.js"
      ],
      "stopAtEntry": false,
      "cwd": "${workspaceFolder}",
      "environment": [],
      "externalConsole": true,
      "MIMode": "lldb"
    }
  ]
}

When switching to debug buildes, I simply change "program" to ${workspaceFolder}/out/Debug/node. The --expose-internals flag is intended to be used by Node.js core developers to access internals from JS land.

V8 has a few convenient GDB macros that call into its internal functions to display useful information about the JS land in debug builds. danbev has a port of those for LLDB available in his learning-v8 repo, I borrow the .lldbinit and lldb_commands.py there and put them in my home directory (so lldb would load them automatically in every debug session). I recommend checking them out if you are also a LLDB user. These macros are also available when you use the debug console of vscode-lldb. There are notes on how to use these macros in the README of learning-v8. You can also check out the article Debug V8 in Node.js core with GDB written by finkelmann, or simply read the comments in deps/v8/tools/gdbinit.

V8 also has some additional GDB integration built-in, like the GDB JIT integration (available with configure --gdb in Node.js) and breakpoint support for embedded builtins, each of them with their own limitations when it comes to platform support, but they can be handy when they happen to fit your use cases.

llnode

llnode is a plugin of LLDB that understands the JS part of the memory (e.g. JS objects on the heap, stack frames) in a Node.js/V8 process.

The nice thing about llnode is that it works on any V8 build with post-mortem support enabled (available through the v8_postmortem_support build flag). By default, Node.js releases are built with post-mortem support enabled, so llnode is particularly handy when debugging a core dump of a Node.js process. The tricks mentioned in the previous section usually only work in debug builds, but if you are running a Node.js server application in production, it’s not usually feasible to run a debug build in order to reproduce a crash. Using llnode, you can debug a crash in production offline as long as you can obtain the core dump and the Node.js binary that generates the core dump.

Another nice thing about llnode is that it supports many platforms - macOS, most Linux distributions, freeBSD, and even Windows (I’ve never tried it on Windows but a few contributors added some tricks to make that work). You can also debug core dumps from another platform with LLDB - for example I usually debug Linux core dumps on macOS.

It’s fairly trivial to install llnode via npm as long as you already have a working lldb installation. The pre-install script tries a few tricks to detect the best lldb installation to build the plugin for and installs an llnode shortcut for you - that’s basically a wrapper of the LLDB console with the plugin loaded, but you can also customize the installation.

1
2
3
npm install -g llnode
llnode /path/to/node/binary -c /path/to/core/dump
(llnode) v8 help  # to see the help text

At the moment, it requires some non-trival effort to make llnode work with a new release of V8 (I gave an introduction to how it works under the hood in this talk), so it’s only reliably usable with Node.js releases - there is no guarantee that it works with the Node.js master branch or canary branch. But when working on bug reports from users (who typically only use releases) or when debugging builds of the release branches, llnode is still quite helpful.

Logging from Node.js Core

Debuggers are handy but sometimes you want logs if you cannot manage to hit the offending code path precisely or if the whole execution flow is too overwhelming. If you are building Node.js locally, it’s pretty easy to print logs to facilitate debugging by modifying the source. There are several ways to do logging from Node.js core:

  1. Put some good old console.log() or console.error() in the JS code that you need to debug, which uses util.inspect underneath - you can use util.inspect.defaultOptions to tweak the options used by these functions globally, for example setting util.inspect.defaultOptions.showHidden = true would print properties that are usually hidden to reduce compatibility burden or verbosity of the output.
  2. Sometimes you may not want to go through the stream abstractions in console.log() - e.g. when you are debugging streams or the console themselves. Then you can try process._rawDebug() which prints a string (and only a string) to stderr directly via a C++ binding.
  3. Of course you can always use printf/fprintf/snprintf etc. in C++ as well.

Note: the global console in Node.js is implemented by Node.js itself in JavaScript on top of process.stdout/process.stderr, with modules like streams, tty, net, and fs. There is another console from V8 which is available through require('inspector').console. Node.js overwrites that with its own implementation during bootstrap.

The utilities mentioned above are usually only added when you are locally developing the project, and the code is usually removed before you commit and submit a PR. There are, however, a few other utilities that we sometimes keep in the release build in order to help collecting useful logs from users to investigate into bug reports:

  1. In JS land, you can use util.debuglog which is similar to debug npm module, though it’s less fancy because it’s in core. In general, doing something like NODE_DEBUG=http node test.js should print the util.debuglog('http')(...) outputs to stderr in the http subsystem (they span across many files e.g. lib/_http_*.js).
  2. In C++ land, you can use Debug() from src/debug_utils.h, which also conditionally output things to stderr based on the values of the NODE_DEBUG environment variable. The DebugCategory there is based on the provider types of AsyncWraps, check out the header and grep for examples for details.

Assertions

You can use the CHECK macros both in C++ land and in JS land, although at the moment these macros are only extensively used in C++. In C++ land there are also DCHECK macros that are only effective in debug builds. Check out tools/check_macros.py (for JS) and src/util.h (for C++) to see what helpers are available.

In JS land you can also use the assert builtin module. The CHECK macros crash (abort) the process immediately whereas assert only throws normal JS exceptions that may be handled later by machineries like domain or the uncaughtException/unhandledRejection events.

Normally I just use CHECK in C++ land and very rarely use assert in JS. The DCHECK macros in C++ were only added recently but I can see myself using them more often as they don’t really cost anything in release builds.

V8 intrinsics

You can also get access to the V8 intrinsics function (prefixed with %) by passing --allow-natives-syntax in the command line to find out more about the internal states in the VM. These functions are not documented - for good reasons, there is zero compatibility guarantee about them - but you can find a list in deps/v8/src/runtime.h. In particular %DebugPrint(obj) can be quite helpful - it prints internal details about a V8 object. Note that %DebugPrint only displays very limited information in a release build - you’ll need a debug build to see more. The global.gc() function exposed by the --expose-gc command line flag that triggers garbage collections can also be pretty handy when debugging memory issues.