Start moving src/crypto functionality out to a separate dep that
can be shared with other projects that need to emulate Node.js
crypto behavior.
PR-URL: https://github.com/nodejs/node/pull/53803
Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com>
Projects that seek to implement Node.js compatible APIs end up
needed to reproduce various bits of functionality internally in
order to faithfully replicate the Node.js behaviors. This is
particularly true for things like byte manipulation, base64 and
hex encoding, and other low-level operations. This change
proposes moving much of this low-level byte manipulation code
out of nodejs/src and into a new `nbytes` library. Initially this
new library will exist in the `deps` directory but the intent is
to spin out a new separate repository to be its home in the future.
Doing so will allow other projects to use the nbytes library with
exactly the same implementation as Node.js.
This commit moves only the byte swapping and legacy base64 handling
code. Additional commits will move additional byte manipulation
logic into the library.
PR-URL: https://github.com/nodejs/node/pull/53507
Reviewed-By: Yagiz Nizipli <yagiz.nizipli@sentry.io>
Reviewed-By: Robert Nagy <ronagy@icloud.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
This commit introduces an experimental implementation of the Web
Storage API using SQLite as the backing data store.
PR-URL: https://github.com/nodejs/node/pull/52435
Reviewed-By: Yagiz Nizipli <yagiz.nizipli@sentry.io>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Ethan Arrowood <ethan@arrowood.dev>
PR-URL: https://github.com/nodejs/node/pull/53124
Reviewed-By: Marco Ippolito <marcoippolito54@gmail.com>
Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/52646
Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>
Reviewed-By: Michael Dawson <midawson@redhat.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Most changes are gated by the `clang==1` condition to avoid breaking
MSVC builds.
Select C/C++ language standard with ClCompile options.
This avoids passing the `-std:c++20` flag while compiling C code.
Do it only under clang option to avoid breaking addons until node-gyp
supports the new LanguageStandard options.
Disable precompiled header configuration for now as it doesn't seem to
work with clang-cl.
Disable C++20 warnings emitted by the Visual Studio C++ STL.
They're very noisy and not our responsibility to fix.
Co-authored-by: Daniel Lemire <daniel@lemire.me>
Co-authored-by: Stefan Stojanovic <stefan.stojanovic@janeasystems.com>
PR-URL: https://github.com/nodejs/node/pull/52870
Reviewed-By: Daniel Lemire <daniel@lemire.me>
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
Fixes: https://github.com/nodejs/node/issues/35339
Co-authored-by: Pooja D P <Pooja.D.P@ibm.com>
Co-authored-by: Teutates <103068388+Teutates@users.noreply.github.com>
Co-authored-by: Antoine du Hamel <duhamelantoine1995@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/43987
Reviewed-By: Michael Dawson <midawson@redhat.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
- Set the clang variable in `config.gypi` so it depends on compiler
checks made by the configure script.
- Replace gyp conditions with `llvm_version` and "0.0" with conditions
that use the `clang` variable.
- Always use `clang==1` or `clang==0` in gyp conditions
PR-URL: https://github.com/nodejs/node/pull/52873
Reviewed-By: Richard Lau <rlau@redhat.com>
Reviewed-By: Yagiz Nizipli <yagiz.nizipli@sentry.io>
Reviewed-By: Mohammed Keyvanzadeh <mohammadkeyvanzade94@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/52609
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Daniel Lemire <daniel@lemire.me>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Stephen Belanger <admin@stephenbelanger.com>
PR-URL: https://github.com/nodejs/node/pull/52719
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Richard Lau <rlau@redhat.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
Reviewed-By: Deokjin Kim <deokjin81.kim@gmail.com>
Reviewed-By: Moshe Atlow <moshe@atlow.co.il>
Reviewed-By: Ulises Gascón <ulisesgascongonzalez@gmail.com>
Reviewed-By: Yagiz Nizipli <yagiz.nizipli@sentry.io>
This allows us to start the profilers before context creation
so that more samples can be collected.
PR-URL: https://github.com/nodejs/node/pull/51783
Reviewed-By: Daniel Lemire <daniel@lemire.me>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
This patch implements automatic on-disk code caching that can be enabled
via an environment variable NODE_COMPILE_CACHE.
When set, whenever Node.js compiles a CommonJS or a ECMAScript Module,
it will use on-disk [V8 code cache][] persisted in the specified
directory to speed up the compilation. This may slow down the first
load of a module graph, but subsequent loads of the same module graph
may get a significant speedup if the contents of the modules do not
change. Locally, this speeds up loading of
test/fixtures/snapshot/typescript.js from ~130ms to ~80ms.
To clean up the generated code cache, simply remove the directory.
It will be recreated the next time the same directory is used for
`NODE_COMPILE_CACHE`.
Compilation cache generated by one version of Node.js may not be used
by a different version of Node.js. Cache generated by different versions
of Node.js will be stored separately if the same directory is used
to persist the cache, so they can co-exist.
Caveat: currently when using this with V8 JavaScript code coverage, the
coverage being collected by V8 may be less precise in functions that are
deserialized from the code cache. It's recommended to turn this off when
running tests to generate precise coverage.
Implementation details:
There is one cache file per module on disk. The directory layout
is:
- Compile cache directory (from NODE_COMPILE_CACHE)
- 8b23c8fe: CRC32 hash of CachedDataVersionTag + NODE_VERESION
- 2ea3424d:
- 10860e5a: CRC32 hash of filename + module type
- 431e9adc: ...
- ...
Inside the cache file, there is a header followed by the actual
cache content:
```
[uint32_t] code size
[uint32_t] code hash
[uint32_t] cache size
[uint32_t] cache hash
... compile cache content ...
```
When reading the cache file, we'll also check if the code size
and code hash match the code that the module loader is loading
and whether the cache size and cache hash match the file content
read. If they don't match, or if V8 rejects the cache passed,
we'll ignore the mismatch cache, and regenerate the cache after
compilation succeeds and rewrite it to disk.
PR-URL: https://github.com/nodejs/node/pull/52535
Refs: https://github.com/nodejs/node/issues/47472
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Yagiz Nizipli <yagiz.nizipli@sentry.io>
Reviewed-By: Mohammed Keyvanzadeh <mohammadkeyvanzade94@gmail.com>
This is similar to the `queryObjects()` console API provided by the
Chromium DevTools console. It can be used to search for objects that
have the matching constructor on its prototype chain in the entire
heap, which can be useful for memory leak regression tests. To avoid
surprising results, users should avoid using this API on constructors
whose implementation they don't control, or on constructors that can
be invoked by other parties in the application.
To avoid accidental leaks, this API does not return raw references to
the objects found. By default, it returns the count of the objects
found. If `options.format` is `'summary'`, it returns an array
containing brief string representations for each object. The visibility
provided in this API is similar to what the heap snapshot provides,
while users can save the cost of serialization and parsing and directly
filer the target objects during the search.
We have been using this API internally for the test suite, which
has been more stable than any other leak regression testing
strategies in the CI. With a public implementation we can now
use the public API instead.
PR-URL: https://github.com/nodejs/node/pull/51927
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Stephen Belanger <admin@stephenbelanger.com>
Reviewed-By: Vinícius Lourenço Claro Cardoso <contact@viniciusl.com.br>
Reviewed-By: Gerhard Stöbich <deb2001-github@yahoo.de>
Reviewed-By: Rafael Gonzaga <rafael.nunu@hotmail.com>
Previously we had two encodings for JS files:
1. If a file contains only ASCII characters, encode it as a one-byte
string (interpreted as uint8_t array during loading).
2. If a file contains any characters with code point above 127,
encode it as a two-byte string (interpreted as uint16_t array
during loading).
This was done because V8 only supports Latin-1 and UTF16 encoding
as underlying representation for strings. To store the JS code
as external strings to save encoding cost and memory overhead
we need to follow the representations supported by V8.
Notice that there is a gap in the Latin1 range (128-255) that we
encoded as two-byte, which was an undocumented TODO for a long
time. That was fine previously because then files that contained
code points beyond the 0-127 range contained code points >255.
Now we have undici which contains code points in the range 0-255
(minus a replaceable code point >255). So this patch adds handling
for the 128-255 range to reduce the size overhead caused by encoding
them as two-byte. This could reduce the size of the binary by
~500KB and helps future files with this kind of code points.
Drive-by: replace `’` with `'` in undici.js to make it a Latin-1
only string. That could be removed if undici updates itself to
replace this character in the comment.
PR-URL: https://github.com/nodejs/node/pull/51605
Reviewed-By: Daniel Lemire <daniel@lemire.me>
Reviewed-By: Ethan Arrowood <ethan@arrowood.dev>
Co-authored-by: Daniel Lemire <daniel@lemire.me>
PR-URL: https://github.com/nodejs/node/pull/50322
Reviewed-By: Jacob Smith <jacob@frende.me>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com>
PR-URL: https://github.com/nodejs/node/pull/50322
Reviewed-By: Jacob Smith <jacob@frende.me>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com>
This makes it easier to locate indeterminism in the snapshot, with
the following command:
$ ./configure --write-snapshot-as-array-literals
$ make V=
$ mv out/Release/obj/gen/node_snapshot.cc ./node_snapshot.cc
$ make V=
$ diff out/Release/obj/gen/node_snapshot.cc ./node_snapshot.cc
PR-URL: https://github.com/nodejs/node/pull/49312
Refs: https://github.com/nodejs/build/issues/3043
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
- Build a static table of octal strings and use it instead of
building octal strings repeatedly during printing.
- Print a newline and an offset for every 64 bytes in the case
of printing array literals so it's easier to locate
variation in snapshot blobs.
- Rework the printing routines so that the differences are only
made in a WriteByteVectorLiteral routine. We can update this
for compression support in the future.
- Rename Snapshot::Generate() that write the data as C++ source
instead of a blob as Snaphost::GenerateAsSource() for clarity,
and move the file stream operations into it to streamline
error handling.
PR-URL: https://github.com/nodejs/node/pull/48851
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
Reviewed-By: Darshan Sen <raisinten@gmail.com>
Previously we hard-code a wrapper id to be used in BaseObjects
to avoid accidentally triggering cppgc on these non-cppgc-managed
objects, but hard-coding can be be hacky and result in mismatch
when we start to create CppHeap ourselves. This patch makes it
more robust by deducing non-cppgc id from the effective cppgc id,
if there is one.
PR-URL: https://github.com/nodejs/node/pull/48660
Refs: 9327503128
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
Reviewed-By: Jiawen Geng <technicalcute@gmail.com>
Incremental compilation of Node.js is slow. Currently on a powerful
Linux machine, it takes about 9 seconds and 830 MB of memory to compile
`gen/node_javascript.cc` with g++. This is the longest step when
recompiling a small change to a Javascript file.
`gen/node_javascript.cc` contains a lot of large binary literals of our
Javascript source code. It is well-known that embedding large binary
literals as C/C++ arrays is slow. One workaround is to include the data
as string literals instead. This is particularly nice for the Javascript
included via js2c, which look better as string literals anyway.
Add a build flag `NODE_JS2C_USE_STRING_LITERALS` to js2c. When this flag
is set, we emit string literals instead of array literals, i.e.:
```c++
// old: static const uint8_t X[] = { ... };
static const uint8_t *X = R"JS2C1b732aee(...)JS2C1b732aee";
// old: static const uint16_t Y[] = { ... };
static const uint16_t *Y = uR"JS2C1b732aee(...)JS2C1b732aee";
```
This requires some modest refactoring in order to deal with the flag
being on or off, but the new code itself is actually shorter.
I only enabled the new flag on Linux/macOS, since those are systems that
I have available for testing. On my Linux system with gcc, it speeds up
compilation by 5.5s (9.0s -> 3.5s). On my Mac system with clang, it
speeds up compilation by 2.2s (3.7s -> 1.5s). (I don't think this flag
will work with MSVC, but it'd probably speed up clang on windows.)
The long-term goal here is probably to allow this to occur incrementally
per Javascript file & in parallel, to avoid recompiling all of
`gen/node_javascript.cc`. Unfortunately the necessary gyp incantations
seem impossible (or at least, far beyond me). Anyway, a 60% speedup is a
nice enough win.
Refs: https://github.com/nodejs/node/issues/47984
PR-URL: https://github.com/nodejs/node/pull/48160
Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com>
Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>
Incremental compilation of Node.js is slow. Currently on a powerful
Linux machine, it takes about 5.8 seconds to compile
`gen/node_snapshot.cc` with g++.
As in the previous PR which dealt with `node_js2c`, we add a new build
define `NODE_MKSNAPSHOT_USE_STRING_LITERALS` which is used by
`node_mksnapshot`. When this flag is set, we emit string literals
instead of array literals for the snapshot blob and for the code cache,
i.e.:
```c++
// old: static const uint8_t X[] = { ... };
static const uint8_t *X = "...";
```
I only enabled the new flag on Linux/macOS, since those are systems that
I have available for testing. On my Linux system with gcc, it speeds up
compilation of this file by 3.7s (5.8s -> 2.1s). On my Mac system with
clang, it speeds up compilation by 1.7s (3.4s -> 1.7s).
Again, the right thing here is probably to generate separate files for
the snapshot blob and for each code cache output, but this is a nice
intermediate speedup.
Refs: https://github.com/nodejs/node/issues/47984
Refs: https://github.com/nodejs/node/pull/48160
PR-URL: https://github.com/nodejs/node/pull/48162
Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com>
Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>
This makes it easier to use third-party dependencies in this tool
(e.g. adding compression using algorithms not available in Python).
It is also much faster - locally js2c.py takes ~1.5s to generate the
output whereas this version takes ~0.1s - and consumes less memory
(~110MB v.s. 66MB).
This also modifies the js2c.py a bit to simplify the output, making
it easier to compare with one generated by the C++ version. Locally
the output from the two are identical. We'll remove js2c.py in a
subsequent commit when the C++ version is used by default.
PR-URL: https://github.com/nodejs/node/pull/46997
Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com>
NodeBIO's memory buffer structure does not support BIO_C_FILE_SEEK and B
IO_C_FILE_TELL. This prevents OpenSSL PEM_read_bio_PrivateKey from readi
ng some private keys. So I switched to OpenSSL'w own protected memory bu
ffers.
Fixes: https://github.com/nodejs/node/issues/47008
PR-URL: https://github.com/nodejs/node/pull/47160
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
Don't link intermediate executables with LTO in order to speed up
overall build time.
Signed-off-by: Konstantin Demin <rockdrilla@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/47313
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Michael Dawson <midawson@redhat.com>
Reviewed-By: Richard Lau <rlau@redhat.com>
Move the bindings used by TextEncoder to a new binding for
more self-contained code.
PR-URL: https://github.com/nodejs/node/pull/46658
Reviewed-By: Darshan Sen <raisinten@gmail.com>
Drive-by: Replace the SFINAE with a static_assert because we don't
have (or need) an implementation for non-scalar AliasedBufferBase
otherwise. Add forward declarations to memory_tracker.h now that
aliased-buffer.h no longer includes util-inl.h.
PR-URL: https://github.com/nodejs/node/pull/46817
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/46579
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com>
Reviewed-By: Robert Nagy <ronagy@icloud.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Darshan Sen <raisinten@gmail.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: Tobias Nießen <tniessen@tnie.de>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Locally the hashing of the binding names sometimes has significant
presence in the profile of bindings, because there can be collisions,
which makes the cost of adding a new binding data non-trivial,
but it's wasteful to spend time on hashing them or dealing with
collisions at all, since we can just use the EmbedderObjectType
enum as the key, as the string names are not actually used beyond
debugging purposes and can be easily matched with a macro.
And since we can just use the enum as the key, we do not even
need the map and can just use an array with the enum as indices
for the lookup.
PR-URL: https://github.com/nodejs/node/pull/46620
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
Python 3.9 on IBM i now properly returns "os400" for sys.platform
instead of claiming to be AIX as it did previously. While the IBM i PASE
environment is compatible with AIX, it is a subset and has numerous
differences which makes it beneficial to distinguish, however this means
that it now needs explicit support here.
PR-URL: https://github.com/nodejs/node/pull/46739
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: Richard Lau <rlau@redhat.com>
See documentation in dataqueue/queue.h for details
Co-authored-by: flakey5 <73616808+flakey5@users.noreply.github.com>
PR-URL: https://github.com/nodejs/node/pull/45258
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/46410
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Tiancheng "Timothy" Gu <timothygu99@gmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Rafael Gonzaga <rafael.nunu@hotmail.com>
Reviewed-By: Robert Nagy <ronagy@icloud.com>
PR-URL: https://github.com/nodejs/node/pull/46410
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Tiancheng "Timothy" Gu <timothygu99@gmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Rafael Gonzaga <rafael.nunu@hotmail.com>
Reviewed-By: Robert Nagy <ronagy@icloud.com>
PR-URL: https://github.com/nodejs/node/pull/45486
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Richard Lau <rlau@redhat.com>
Reviewed-By: Jiawen Geng <technicalcute@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/46194
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Tobias Nießen <tniessen@tnie.de>
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
Reviewed-By: Richard Lau <rlau@redhat.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: Darshan Sen <raisinten@gmail.com>
Remove the dependency on ICU for this part, as well as the
hacky way of converting embedder main sources to UTF-8 via
V8 APIs. Allow `UnionBytes` to own the memory its pointing
to in order to simplify the code on the `BuiltinLoader` side.
PR-URL: https://github.com/nodejs/node/pull/46119
Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Rafael Gonzaga <rafael.nunu@hotmail.com>
Reviewed-By: Darshan Sen <raisinten@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/45803
Reviewed-By: Robert Nagy <ronagy@icloud.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Michael Dawson <midawson@redhat.com>
Generalize the finalizer's second pass callback to make it
cancellable and simplify the code around the second pass callback.
With this change, it is determined that Reference::Finalize or
RefBase::Finalize are called once, either from the env's shutdown,
or from the env's second pass callback.
All existing node-api js tests should pass without a touch. The
js_native_api cctest is no longer applicable with this change,
just removing it.
PR-URL: https://github.com/nodejs/node/pull/44141
Refs: https://github.com/nodejs/node/issues/44071
Reviewed-By: Michael Dawson <midawson@redhat.com>
PR-URL: https://github.com/nodejs/node/pull/45209
Reviewed-By: Richard Lau <rlau@redhat.com>
Reviewed-By: Rafael Gonzaga <rafael.nunu@hotmail.com>
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com>
Add back the Windows resource file to the build scripts that was
mistakenly removed by aa3a572e6b.
PR-URL: https://github.com/nodejs/node/pull/45042
Fixes: https://github.com/nodejs/node/issues/45025
Refs: https://github.com/nodejs/node/pull/43652
Reviewed-By: Michaël Zasso <targos@protonmail.com>
Reviewed-By: Michael Dawson <midawson@redhat.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Rafael Gonzaga <rafael.nunu@hotmail.com>
Reviewed-By: Jiawen Geng <technicalcute@gmail.com>
Reviewed-By: Mohammed Keyvanzadeh <mohammadkeyvanzade94@gmail.com>
Reviewed-By: Darshan Sen <raisinten@gmail.com>
Refs: https://github.com/nodejs/node/issues/44000
- add infra to support externally shared js builtins in
support of distos that want to externalize deps that
include JS/WASM instead of native code
- add support for externalizing
- cjs_module_lexer/lexer
- cjs_module_lexer/dist/lexer
- undici/undici
Signed-off-by: Michael Dawson <mdawson@devrus.com>
PR-URL: https://github.com/nodejs/node/pull/44376
Reviewed-By: Gireesh Punathil <gpunathi@in.ibm.com>
Add an ExitCode enum class and use it throughout the code base
instead of hard-coding the exit codes everywhere. At the moment,
the exit codes used in many places do not actually conform to
what the documentation describes. With the new enums (which
are also available to the JS land as constants in an internal
binding) we could migrate to a more consistent usage of the
codes, and eventually expose the constants to the user land
when they are stable enough.
PR-URL: https://github.com/nodejs/node/pull/44746
Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>
Reviewed-By: Jacob Smith <jacob@frende.me>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Darshan Sen <raisinten@gmail.com>
Commit dae283d96f from August 2020 introduced a call to EntropySource()
in SecretKeyGenTraits::DoKeyGen() in src/crypto/crypto_keygen.cc. There
are two problems with that:
1. It does not check the return value, it assumes EntropySource() always
succeeds, but it can (and sometimes will) fail.
2. The random data returned byEntropySource() may not be
cryptographically strong and therefore not suitable as keying
material.
An example is a freshly booted system or a system without /dev/random or
getrandom(2).
EntropySource() calls out to openssl's RAND_poll() and RAND_bytes() in a
best-effort attempt to obtain random data. OpenSSL has a built-in CSPRNG
but that can fail to initialize, in which case it's possible either:
1. No random data gets written to the output buffer, i.e., the output is
unmodified, or
2. Weak random data is written. It's theoretically possible for the
output to be fully predictable because the CSPRNG starts from a
predictable state.
Replace EntropySource() and CheckEntropy() with new function CSPRNG()
that enforces checking of the return value. Abort on startup when the
entropy pool fails to initialize because that makes it too easy to
compromise the security of the process.
Refs: https://hackerone.com/bugs?report_id=1690000
Refs: https://github.com/nodejs/node/pull/35093
ClientHelloParser::ParseHeader(data, avail) potentially accesses data
beyond avail bytes because it trusts the client to transmit a valid
frame length. Sending an impossibly small frame length causes the TLS
server to read beyond the buffer provided by the caller.
Guard against this by calling End() on the ClientHelloParser when the
client transmits an impossibly small frame length.
The test is designed to reliable cause a segmentation fault on Linux and
Windows when the buffer overrun occurs, and to trigger a spatial safety
violation when compiled with an address sanitizer enabled or when
running under valgrind.
PR-URL: https://github.com/nodejs/node/pull/44580
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
Reviewed-By: Rich Trott <rtrott@gmail.com>
Reviewed-By: Minwoo Jung <nodecorelab@gmail.com>
To distinguish per-context values from the node::Environment, split
those values to a new node::Realm structure and consolidate
bootstrapping methods with it.
PR-URL: https://github.com/nodejs/node/pull/44179
Refs: https://github.com/nodejs/node/issues/42528
Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Allows APM vendors to generate a diagnostic report without calling into
JavaScript. Like, from their own message channels interrupting the
isolate and generating a report on demand.
PR-URL: https://github.com/nodejs/node/pull/44255
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Gireesh Punathil <gpunathi@in.ibm.com>
The term "native module" dates back to some of the oldest code
in the code base. Within the context of Node.js core it usually
refers to modules that are native to Node.js (e.g. fs, http),
but it can cause confusion for people who don't work on this
part of the code base, as "native module" can also refer to
native addons - which is even the case in some of the API
docs and error messages.
This patch tries to make the usage of these terms more consistent.
Now within the context of Node.js core:
- JavaScript scripts that are built-in to Node.js are now referred
to as "built-in(s)". If they are available as modules,
they can also be referred to as "built-in module(s)".
- Dynamically-linked shared objects that are loaded into
the Node.js processes are referred to as "addons".
We will try to avoid using the term "native modules" because it could
be ambiguous.
Changes in this patch:
File names:
- node_native_module.h -> node_builtins.h,
- node_native_module.cc -> node_builtins.cc
C++ binding names:
- `native_module` -> `builtins`
`node::Environment`:
- `native_modules_without_cache` -> `builtins_without_cache`
- `native_modules_with_cache` -> `builtins_with_cache`
- `native_modules_in_snapshot` -> `builtins_in_cache`
- `native_module_require` -> `builtin_module_require`
`node::EnvSerializeInfo`:
- `native_modules` -> `builtins
`node::native_module::NativeModuleLoader`:
- `native_module` namespace -> `builtins` namespace
- `NativeModuleLoader` -> `BuiltinLoader`
- `NativeModuleRecordMap` -> `BuiltinSourceMap`
- `NativeModuleCacheMap` -> `BuiltinCodeCacheMap`
- `ModuleIds` -> `BuiltinIds`
- `ModuleCategories` -> `BuiltinCategories`
- `LoadBuiltinModuleSource` -> `LoadBuiltinSource`
`loader.js`:
- `NativeModule` -> `BuiltinModule` (the `NativeModule` name used in
`process.moduleLoadList` is kept for compatibility)
And other clarifications in the documentation and comments.
PR-URL: https://github.com/nodejs/node/pull/44135
Fixes: https://github.com/nodejs/node/issues/44036
Reviewed-By: Jacob Smith <jacob@frende.me>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Michael Dawson <midawson@redhat.com>
Reviewed-By: Richard Lau <rlau@redhat.com>
Reviewed-By: Jiawen Geng <technicalcute@gmail.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
Reviewed-By: Mohammed Keyvanzadeh <mohammadkeyvanzade94@gmail.com>
Reviewed-By: Tobias Nießen <tniessen@tnie.de>
Reviewed-By: Jan Krems <jan.krems@gmail.com>
Now that we include the code cache into the embedded snapshot,
there is no point in splitting an Environment-independent
NativeModuleLoader out of NativeModuleEnv. Merge the two
classes for simplicity.
PR-URL: https://github.com/nodejs/node/pull/43824
Refs: https://github.com/nodejs/node/issues/31074
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
There are no clear indicators anyone is using the dtrace USDT probes.
ETW support is very intertwined with the dtrace infrastructure. It's not
clear if anyone uses ETW so to keep things simple it too is removed.
Fixes: https://github.com/nodejs/node/issues/43649
PR-URL: https://github.com/nodejs/node/pull/43652
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Michaël Zasso <targos@protonmail.com>
Reviewed-By: Santiago Gimeno <santiago.gimeno@gmail.com>
Reviewed-By: Juan José Arboleda <soyjuanarbol@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Gerhard Stöbich <deb2001-github@yahoo.de>
This commit adds the setting of an appname (configuration section
name), 'nodejs_conf', to be used when reading OpenSSL configuration
files.
The motivation for this is that currently the default OpenSSL
configuration, 'openssl_conf', element will be used which may be
undesirable as it might configure OpenSSL in unwanted ways. With this
commit it is still possible to use a default openssl.cnf file but the
only section that Node.js will read from is a section named
'nodejs_conf'.
PR-URL: https://github.com/nodejs/node/pull/43124
Refs: https://github.com/nodejs/node/issues/40366
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Rich Trott <rtrott@gmail.com>
Reviewed-By: Rafael Gonzaga <rafael.nunu@hotmail.com>
Reviewed-By: Beth Griggs <bgriggs@redhat.com>
Since V8 code cache encodes indices to the read-only space
it is safer to make sure that the code cache is generated in the
same heap used to generate the embdded snapshot. This patch
merges the code cache builder into the snapshot builder and
makes the code cache part of node::SnapshotData that is
deserialized into the native module loader during bootstrap.
PR-URL: https://github.com/nodejs/node/pull/43023
Fixes: https://github.com/nodejs/node/issues/31074
Refs: https://github.com/nodejs/node/issues/35711
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
Since all its uses are now gone, it's time to say goodbye to
AllocatedBuffer.
Refs: https://github.com/nodejs/node/pull/39941
Signed-off-by: Darshan Sen <raisinten@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/43008
Reviewed-By: Tobias Nießen <tniessen@tnie.de>
Reviewed-By: Rich Trott <rtrott@gmail.com>
Reviewed-By: Mohammed Keyvanzadeh <mohammadkeyvanzade94@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
Reviewed-By: Zeyu "Alex" Yang <himself65@outlook.com>
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Node.js unofficially supports a shared library variant where the
main node executable is a thin wrapper around node.dll/libnode.so.
The key benefit of this is to support embedding Node.js in other
applications.
Since Node.js 12 there have been a number of issues preventing the
shared library build from working correctly, primarily on Windows:
* A number of functions used executables such as `mksnapshot` are
not exported from `libnode.dll` using a `NODE_EXTERN` attribute
* A dependency on the `Winmm` system library is missing
* Incorrect defines on executable targets leads to `node.exe`
claiming to export a number of functions that are actually in
`libnode.dll`
* Because `node.exe` attempts to export symbols, `node.lib` gets
generated causing native extensions to try to link against
`node.exe` not `libnode.dll`.
* Similarly, because `node.dll` was renamed to `libnode.dll`,
native extensions don't know to look for `libnode.lib` rather
than `node.lib`.
* On macOS an RPATH is added to find `libnode.dylib` relative to
`node` in the same folder. This works fine from the
`out/Release` folder but not from an installed prefix, where
`node` will be in `bin/` and `libnode.dylib` will be in `lib/`.
* Similarly on Linux, no RPATH is added so LD_LIBRARY_PATH needs
setting correctly for `bin/node` to find `lib/libnode.so`.
For the `libnode.lib` vs `node.lib` issue there are two possible
options:
1. Ensure `node.lib` from `node.exe` does not get generated, and
instead copy `libnode.lib` to `node.lib`. This means addons
compiled when referencing the correct `node.lib` file will
correctly depend on `libnode.dll`. The down side is that
native addons compiled with stock Node.js will still try to
resolve symbols against node.exe rather than libnode.dll.
2. After building `libnode.dll`, dump the exports using `dumpbin`,
and process this to generate a `node.def` file to be linked into
`node.exe` with the `/DEF:node.def` flag. The export entries
in `node.def` will all read
```
my_symbol=libnode.my_symbol
```
so that `node.exe` will redirect all exported symbols back to
`libnode.dll`. This has the benefit that addons compiled with
stock Node.js will load correctly into `node.exe` from a shared
library build, but means that every embedding executable also
needs to perform this same trick.
I went with the first option as it is the cleaner of the two
solutions in my opinion. Projects wishing to generate a shared
library variant of Node.js can now, for example,
```
.\vcbuild dll package vs
```
to generate a full node installation including `libnode.dll`,
`Release\node.lib`, and all the necessary headers. Native addons
can then be built against the shared library build easily by
specifying the correct `nodedir` option.
For example
```
>npx node-gyp configure --nodedir
C:\Users\User\node\Release\node-v18.0.0-win-x64
...
>npx node-gyp build
...
>dumpbin /dependents build\Release\binding.node
Microsoft (R) COFF/PE Dumper Version 14.29.30136.0
Copyright (C) Microsoft Corporation. All rights reserved.
Dump of file build\Release\binding.node
File Type: DLL
Image has the following dependencies:
KERNEL32.dll
libnode.dll
VCRUNTIME140.dll
api-ms-win-crt-string-l1-1-0.dll
api-ms-win-crt-stdio-l1-1-0.dll
api-ms-win-crt-runtime-l1-1-0.dll
...
```
PR-URL: https://github.com/nodejs/node/pull/41850
Reviewed-By: Michael Dawson <midawson@redhat.com>
Reviewed-By: Beth Griggs <bgriggs@redhat.com>
Reviewed-By: Richard Lau <rlau@redhat.com>
So that the embedded snapshot can be reused by the worker.
PR-URL: https://github.com/nodejs/node/pull/42702
Refs: https://github.com/nodejs/node/issues/35711
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Tobias Nießen <tniessen@tnie.de>
Reviewed-By: James M Snell <jasnell@gmail.com>
This adds a --build-snapshot runtime option which is currently only
supported by the node_mksnapshot binary, and a --node-snapshot-main
configure option that makes use it to run a custom script when
building the embedded snapshot. The idea is to have this experimental
feature in core as a configure-time feature for now, and investigate
the renaming V8 bugs before we make it available to more users via
making it a runtime option.
PR-URL: https://github.com/nodejs/node/pull/42466
Reviewed-By: Darshan Sen <raisinten@gmail.com>
Reviewed-By: Mohammed Keyvanzadeh <mohammadkeyvanzade94@gmail.com>
Reviewed-By: Khaidi Chu <i@2333.moe>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
