Memory leak regression testing with V8/Node.js, part 3 - heap iteration-based testing

In the previous blog post, I described the heap snapshot trick as an “abuse” of the heap snapshot API, because heap snapshots are not designed to interact with the finalizers run in the heap. But the concept of using heap snapshots to identify/disprove leak itself is not an abuse - that’s exactly what they are designed for. Well then, can we use heap snapshots, or a simpler version of it, to test gainst memory leaks?

Chrome DevTools console API

Technically, we can already do the testing using existing APIs and there is no need to use the finalizers. Just take two heap snapshots before and after a certain amount of allocation, and find the difference in them to learn whether certain kind of objects can be garbage collected - this is an intended & documented use case of heap snapshots. It’s just that using finalizers to monitor specific objects is simpler and faster than parsing & diffing the heap snapshots generated, so we took the shortcut here. But what if we can do the diffing without generating a heap snapshot at all?

Turns out there is already a console API in Chromium’s DevTools that allows a similar use case - queryObject(constructor), which does a pretty aggressive garbage collection which is the same one used by heap snapshots, and searches the current execution context to find objects whose prototype chain contains that constructor’s prototype.

// In the DevTools console:

class A {}
const a = new A();
// Returns undefined, then logs an array that is just [a]
queryObjects(A);

Internal API in Node.js and new test helpers

Since this is already implemented in V8, I just added a v8::HeapProfiler API that allows embedders to do a similar search and collect references to objects that meets a customized predicate.

Now that we have a new V8 API for searching objects on the heap after an aggressive GC, the next step is to devise a memory leak testing strategy with it.

First, a countObjectsWithPrototype(prototype) helper was added to Node.js’s internal binding (we were only using it for Node.js’s own tests, to check this strategy is good enough as a replacement). This is similar to queryObjects() except that it takes the prototype directly, and returns the count of the objects found because that’s all what we needed for the test.

class A {}
const a = new A();
countObjectsWithPrototype(A.prototype);  // 1

Using the new internal API, a naive memory leak checker could be implemented like this:

async function checkIfCollectableByCounting(fn, klass, count) {
  const initialCount = countObjectsWithPrototype(klass.prototype);

  for (let i = 0; i < count; ++i) {
    // Here, fn() should create one and only one object
    // of klass.
    const obj = await fn(i);
  }
  const remainingCount = countObjectsWithPrototype(klass.prototype);
  const collected = initialCount + count - remainingCount;
  if (collected > 0) {
     console.log(`${klass.name} is collectable (${collected} collected)`);
     return;
  }
  throw new Error(`${klass.name} cannot be collected`);
}

// Usage:
const leakMe = [];
class A { constructor() { leakMe.push(this); } }
function factory() { return new A; }
// It will throw because all the A created are put into an array
// that's still alive and therefore not collectable.
checkIfCollectableByCounting(factory, A, 1000);

But when I tried to use this strategy to deflake a particularly brittle memory leak test, it didn’t work quite well: if the graph produced by fn() is full of weak referenes, and the object creation loop grows the heap too fast, once we limit the heap size to a smaller value, the last resort GC performed by V8 is not effective enough to save the application from running out of memory before we get to the second countObjectsWithPrototype() call. Then again we are running into false positives (in normal applications, the growth shouldn’t be this fast because users rarely create this amount of weak references in a hot loop).

Now, it may be tempting to just count in each iteration of the object creation loop and finish the test early once we detect that any of the objects creaetd are collected. But since checkIfCollectableByCounting() incurs GC and iterate over the heap, it takes quite a bit of time and can slow the test down significantly.

After some tweaking, adding these following tricks seemed to make the test work reliably enough:

1. Instead of creating just one object per iteration and create all of them in just one loop, create them in batches. This makes the heap grow fast enough so that the test can finish quickly, but also not too fast so that V8’s GC cannot keep up when the graph produced is too complex to GC efficiently.
2. Give the GC a bit of time to do its work after every batch
3. Check if any objects are collected after each batch, instead of waiting until all the objects are created before checking. This again helps to make the test finish faster.

So the final version looks like this:

const wait = require('timers/promises').setTimeout;

async function checkIfCollectableByCounting(fn, klass, count, waitTime = 20) {
  const initialCount = countObjectsWithPrototype(klass.prototype);

  let totalCreated = 0;
  for (let i = 0; i < count; ++i) {
    // Here, fn() create the objects in batches, and return the count
    // of objects created in the batch.
    const created = await fn(i);
    totalCreated += created;
    await wait(waitTime);  // give GC some breathing room.

    // If we already find some collected objects after
    // processing a batch, it's good enough to stop.
    const currentCount = countObjectsWithPrototype(klass.prototype);
    const collected = initialCount + totalCreated - currentCount;

    if (collected > 0) {
      console.log(`Detected ${collected} collected ${name}, finish early`);
      return;
    }
  }

  // Final check.
  await wait(waitTime);  // give GC some breathing room.
  const currentCount = countObjectsWithPrototype(klass.prototype);
  const collected = initialCount + totalCreated - currentCount;

  if (collected > 0) {
    console.log(`Detected ${collected} collected ${klass.name}`);
    return;
  }

  throw new Error(`${klass.name} cannot be collected`);
}

// Usage:
const leakMe = [];
class A { constructor() { leakMe.push(this); } }
function factory() {
  for (let i = 0; i < 100; ++i) new A;
  return 100;
}
// It will throw because all the A created are put into an array
// that's still alive and therefore not collectable.
checkIfCollectableByCounting(factory, A, 10);

New public Node.js API: v8.queryObjects()

The final version above has worked pretty good so far - no more flakes in the CI. It does seem to be be something that can be useful to Node.js users too, though, based on my conversations with others about it. So I opened a pull request to expose this to the v8 built-in library in Node.js.

The public API behaves more like the Chrome DevTools console API, except that it does return information instead of only logging them. To avoid accidentally leaking references to objects found, the API doesn’t return references. By default it returns the count, which is usually enough for testing. To help debugging, it can also return string summaries for the objects found.

const { queryObjects } = require('v8');
class A { foo = 'bar'; }
console.log(queryObjects(A)); // 0
const a = new A();
console.log(queryObjects(A)); // 1
// [ "A { foo: 'bar' }" ]
console.log(queryObjects(A, { format: 'summary' }));

Since this API is based on prototype walking, similar to the Chrome DevTools API, be aware that if the class is inherited, the child class’ prototype chain contains the prototype of the constructor passed into the API, so the child class’ prototype is considered a match.

class B extends A { bar = 'qux'; }
const b = new B();
console.log(queryObjects(B)); // 1
// [ "B { foo: 'bar', bar: 'qux' }" ]
console.log(queryObjects(B, { format: 'summary' }));

console.log(queryObjects(A));  // 3
// [ "B { foo: 'bar', bar: 'qux' }", 'A {}', "A { foo: 'bar' }" ]
console.log(queryObjects(A, { format: 'summary' }));

Using the new API, the helper above can just change countObjectsWithPrototype(klass.prototype) to v8.queryObjects(klass) and no longer relies on an internal API to work. Hopefully, this new API can help others make their memory leak regression tests produce fewer false positives too.

Final remarks

Thanks Bloomberg for sponsoring my memory-related work in Node.js.