Node.js native addon without node-gyp

🌐 - 简体中文

Contents

• Why explore more way for native code binding?

• Compile addons manually on Linux and macOS.

• Limits and workaround on Windows.

• You may not need NAPI. For example, ESBuild does a special way.

Why

It's certain that we need to bind native code for some compute-intensive tasks. Nowadays, NAPI (node-api) + node-gyp is the most popular pair for Node.js native addons.

But due to historical reasons node-gyp is really sucks using node-gyp can be difficult. Some of the disadvantages of node-gyp include that it can easily result in problems that are hard to resolve. And the gyp, which node-gyp based on, is more opaque and less used by normal C / C++ developers. So why not use our general-purpose build tools instead of the node-gyp?

Build an Addon Manually

git clone https://github.com/kkocdko/hinapi

NAPI is a set of ABI stable functions, include the node_api.h the crux of the matter. And here we want to use node-addon-api (c++), certainly, you can use only C.

The easiest way if you already have CMake installed (don't run away! We will soon mention no cmake method):

# in ./hinapi
cmake -B build
cmake --build build
node src/main.js
du third_party -sh

Outputs:

[kkocdko@klf hinapi]$ cmake -B build
-- The C compiler identification is GNU 12.2.1
...
added 2 packages in 624ms
...
-- Build files have been written to: /home/kkocdko/misc/code/hinapi/build

[kkocdko@klf hinapi]$ cmake --build build
[ 50%] Building CXX object CMakeFiles/hinapi.dir/src/hinapi.cc.o
[100%] Linking CXX shared library hinapi.node
[100%] Built target hinapi

[kkocdko@klf hinapi]$ node src/main.js
calc 1 + 2 = 3
created object = { name: 'tom', age: 'tom' }
callback argument = hello world
promise resolved value = 1.2

[kkocdko@klf hinapi]$ du third_party -sh
508K    third_party

The CMakeLists.txt in template repo is "just works", supports MSVC, GCC, Clang, on Linux, Windows (MSVC and MinGW), and macOS. See this repo's GitHub Actions.

If you wouldn't want CMake, I can tell you what CMakeLists.txt do:

1. Download node-api-headers (currently, it's from Node.js 19) and node-addon-api (c++) to ./third_party.

2. What we want is hinapi.node, a dynamic library. On Linux, run g++ src/hinapi.cc -o build/hinapi.node -I third_party/node-addon-api -I third_party/node-api-headers/include -shared -fPIC.

3. However, Windows require a DLL to resolve all symbols at linking. so we need a .def file which defined the exported NAPI functions, and use lib (MSVC) / dlltool (MinGW) to create libnode.lib (MSVC) / libnode.a (MinGW), then link the static libs. Thank goodness this is much easier on macOS, just add the -undefined dynamic_lookup.

Pros: 1. Lightweight, less than 0.6 MiB dependencies. 2. Transparent, fully control your compilation process.

That's all! Now, try node src/main.js.

Do You Really Need NAPI?

Let's go back to the beginning:

It's certain that we need to bind native code for some compute-intensive tasks.

Is this really true? Not always. For some workloads, you can try the following methods to bring native speed to your Node.js app.

C FFI

Wouldn't want advanced features like async, promise and complex JavaScript objects? Call dlopen() and just invoke dynamic libs' exported functions.

This function is also implemented in both Bun and Deno, you'll able to run your app on these runtimes less painful.

By Stdio

There's a slogan in Effective Go:

Do not communicate by sharing memory; instead, share memory by communicating.

ESBuild, a bundler for web, explain this for us perfectly! It use stdio to communicate between ESBuild process and Node.js process. Not only, but many famous projects like LSP use this.

You may doubt it's performance, however, in many cases, the bottleneck of stdio is terminal, not shell / program. On my machine, stdio takes 3x time compare to memcpy while transport same data.

The pros are simpler and better compatibility (more languages' invoking support), yeah, stdio is one of the most common and initial IPC methods.

But the cons are not to be overlooked. You need to process and wrap data on both ends, an extra process is needed, causing bigger latency and heavier memory footprint.

End

Hoping this post will give you more options when you need native performance in Node.js apps. Thanks for every project / link mentioned above.

Translation

git clone https://github.com/kkocdko/hinapi

# 在目录 ./hinapi
cmake -B build
cmake --build build
node src/main.js
du third_party -sh

[kkocdko@klf hinapi]$ cmake -B build
-- The C compiler identification is GNU 12.2.1
...
added 2 packages in 624ms
...
-- Build files have been written to: /home/kkocdko/misc/code/hinapi/build

[kkocdko@klf hinapi]$ cmake --build build
[ 50%] Building CXX object CMakeFiles/hinapi.dir/src/hinapi.cc.o
[100%] Linking CXX shared library hinapi.node
[100%] Built target hinapi

[kkocdko@klf hinapi]$ node src/main.js
calc 1 + 2 = 3
created object = { name: 'tom', age: 'tom' }
callback argument = hello world
promise resolved value = 1.2

[kkocdko@klf hinapi]$ du third_party -sh
508K    third_party