Tips and Tricks for Node.js Core Development and Debugging
I thought about writing some guides on this topic in the nodejs/node repo, but it’s easier to throw whatever tricks I personally use on the Internet first - I am also going to heavily use the pronouns “I”, “We” and “You” in this post, and talk about my personal preference here, both of which we always try to avoid in the repo ;)
Note that many hidden features mentioned in this post are not meant to be used in the user land of Node.js, and there is zero stability guarantee about them. Do not rely on them in any project outside the scope of Node.js core.
You can skip this section if you already know about the basic structure of the Node.js source code and how to build it.
lib directory and the C++ part is mostly under the
src directory. There are also a bunch of third-party dependencies like v8 or libuv which are placed under the
deps directory. This post covers the techniques on how to debug most of them. There are also things like Python scripts and DTrace scripts in the code base but I don’t usually need to (or, I don’t really know how to) debug them with anything fancy, so I won’t cover those in this post.
The Node.js release binaries do not strip the debug symbols (surprised?) so you don’t necessarily have to build a debug build to get meaningful C/C++ stack traces or to do step debugging in a native debugger like LLDB or GDB. That said the release binary is built with
-Os so not everything is available in the release build, and you do need to use a debug build if you want to use V8’s internal debug utilities.
The debug build can be configured by adding
--debug when you run the
configure script before building Node.js (on non-Windows systems at least, the
vcbuild.bat script we use on Windows runs
configure for you), and will be built as
out/Debug/node (symlinked as
./node_g in the project directory). So typically this is what I do:
# For release builds
For more information check out BUILDING.md in the project directory.
The C++ intellisense offered by the cpptools plugin can consume a compilation command database for better understanding of the code base and the build flags that you are using. To generate the database for Node.js, add a
-C switch when you run
configure. For example, to generate the database for the release build, run
in the project directory, and in
to your configurations. Or do
./configure --debug -C and replace
Debug in the snippet above if you are using debug builds.
On how to integrate the plugin itself in VSCode, see the docs of the plugin for details.
This is what I tend to use to debug in the JS land these days because it works with most Node.js binaries I need to debug.
Usually I would write a test script in a local file, say,
test.js, to trigger the code I want to execute, and run
./node --inspect-brk test.js
Then I usually open up Chrome, type
chrome://inspect in the address bar, and use the Node.js dedicated DevTools UI, where the execution should stop at the first line of
test.js - it is possible to use other clients like Visual Studio Code or
node inspect that is bundled into the binary, but my personal reference is to use Chrome to debug JS and use VSCode to debugger C++ (more on that later). Note that with certain combinations of Node.js and Chrome certain features may be broken in the inspector, and sometimes you’ll need to use the Chrome Canary to get that fix early before it enters the stable channel of Chrome (example).
Sometimes when I need to debug into the bootstrap process of Node.js, I use the hidden switch introduced by danbev for Node.js core developers:
./node --inspect-brk-node test.js
Which is supposed to break into the first line of the bootstrapping JS code of Node.js, currently this is implemented as an explicit
debugger; statement in
lib/internal/bootstrap/loaders.js. Theoretically that’s not the first JS script we execute during bootstrap (that would be
lib/internal/per_context.js at the time this post was written), but it’s the first JS script that’s executed after we bootstrap to a point where the inspector can be started.
If you want to set a breakpoint programmatically, adding the
debugger; statement in the JS code would do the trick. Note that you need to recompile the binary to see the changes because the builtin JS code in Node.js are compiled into the binary as static C char arrays instead of being loaded from disk. Also, remember to remove these statements before you commit and send a PR.
There are only two debuggers I personally use: gdb (on Linux) and lldb (on macOS and sometimes Linux). I don’t usually develop Node.js core on Windows but the debugger in Visual Studio should also work just fine, though I am not aware of any special tricks that can be used in VS.
I typically use the vscode-lldb plugin of Visual Studio Code for daily development. This is what my
launch.json usually looks like
When switching to debug buildes, I simply change
--expose-internals flag is intended to be used by Node.js core developers to access internals from JS land.
V8 has a few convenient GDB macros that call into its internal functions to display useful information about the JS land in debug builds. danbev has a port of those for LLDB available in his learning-v8 repo, I borrow the
lldb_commands.py there and put them in my home directory (so lldb would load them automatically in every debug session). I recommend checking them out if you are also a LLDB user. These macros are also available when you use the debug console of vscode-lldb. There are notes on how to use these macros in the README of learning-v8. You can also check out the article Debug V8 in Node.js core with GDB written by finkelmann, or simply read the comments in
V8 also has some additional GDB integration built-in, like the GDB JIT integration (available with
configure --gdb in Node.js) and breakpoint support for embedded builtins, each of them with their own limitations when it comes to platform support, but they can be handy when they happen to fit your use cases.
llnode is a plugin of LLDB that understands the JS part of the memory (e.g. JS objects on the heap, stack frames) in a Node.js/V8 process.
The nice thing about llnode is that it works on any V8 build with post-mortem support enabled (available through the
v8_postmortem_support build flag). By default, Node.js releases are built with post-mortem support enabled, so llnode is particularly handy when debugging a core dump of a Node.js process. The tricks mentioned in the previous section usually only work in debug builds, but if you are running a Node.js server application in production, it’s not usually feasible to run a debug build in order to reproduce a crash. Using llnode, you can debug a crash in production offline as long as you can obtain the core dump and the Node.js binary that generates the core dump.
Another nice thing about llnode is that it supports many platforms - macOS, most Linux distributions, freeBSD, and even Windows (I’ve never tried it on Windows but a few contributors added some tricks to make that work). You can also debug core dumps from another platform with LLDB - for example I usually debug Linux core dumps on macOS.
It’s fairly trivial to install llnode via npm as long as you already have a working lldb installation. The pre-install script tries a few tricks to detect the best lldb installation to build the plugin for and installs an
llnode shortcut for you - that’s basically a wrapper of the LLDB console with the plugin loaded, but you can also customize the installation.
npm install -g llnode
At the moment, it requires some non-trival effort to make llnode work with a new release of V8 (I gave an introduction to how it works under the hood in this talk), so it’s only reliably usable with Node.js releases - there is no guarantee that it works with the Node.js master branch or canary branch. But when working on bug reports from users (who typically only use releases) or when debugging builds of the release branches, llnode is still quite helpful.
Debuggers are handy but sometimes you want logs if you cannot manage to hit the offending code path precisely or if the whole execution flow is too overwhelming. If you are building Node.js locally, it’s pretty easy to print logs to facilitate debugging by modifying the source. There are several ways to do logging from Node.js core:
- Put some good old
console.error()in the JS code that you need to debug, which uses
util.inspectunderneath - you can use
util.inspect.defaultOptionsto tweak the options used by these functions globally, for example setting
util.inspect.defaultOptions.showHidden = truewould print properties that are usually hidden to reduce compatibility burden or verbosity of the output.
- Sometimes you may not want to go through the stream abstractions in
console.log()- e.g. when you are debugging streams or the console themselves. Then you can try
process._rawDebug()which prints a string (and only a string) to stderr directly via a C++ binding.
- Of course you can always use
snprintfetc. in C++ as well.
Note: the global
process.stderr, with modules like streams, tty, net, and fs. There is another
console from V8 which is available through
require('inspector').console. Node.js overwrites that with its own implementation during bootstrap.
The utilities mentioned above are usually only added when you are locally developing the project, and the code is usually removed before you commit and submit a PR. There are, however, a few other utilities that we sometimes keep in the release build in order to help collecting useful logs from users to investigate into bug reports:
- In JS land, you can use
util.debuglogwhich is similar to
debugnpm module, though it’s less fancy because it’s in core. In general, doing something like
NODE_DEBUG=http node test.jsshould print the
util.debuglog('http')(...)outputs to stderr in the http subsystem (they span across many files e.g.
- In C++ land, you can use
src/debug_utils.h, which also conditionally output things to stderr based on the values of the
NODE_DEBUGenvironment variable. The
DebugCategorythere is based on the provider types of
AsyncWraps, check out the header and grep for examples for details.
You can use the
CHECK macros both in C++ land and in JS land, although at the moment these macros are only extensively used in C++. In C++ land there are also
DCHECK macros that are only effective in debug builds. Check out
tools/check_macros.py (for JS) and
src/util.h (for C++) to see what helpers are available.
In JS land you can also use the
assert builtin module. The
CHECK macros crash (abort) the process immediately whereas
assert only throws normal JS exceptions that may be handled later by machineries like domain or the
Normally I just use
CHECK in C++ land and very rarely use
assert in JS. The
DCHECK macros in C++ were only added recently but I can see myself using them more often as they don’t really cost anything in release builds.
You can also get access to the V8 intrinsics function (prefixed with
%) by passing
--allow-natives-syntax in the command line to find out more about the internal states in the VM. These functions are not documented - for good reasons, there is zero compatibility guarantee about them - but you can find a list in
deps/v8/src/runtime.h. In particular
%DebugPrint(obj) can be quite helpful - it prints internal details about a V8 object. Note that
%DebugPrint only displays very limited information in a release build - you’ll need a debug build to see more. The
global.gc() function exposed by the
--expose-gc command line flag that triggers garbage collections can also be pretty handy when debugging memory issues.