Support `.mjs` output [also a TypeScript issue]

[SMotaal]

I guess we are kind of being forced a little here but someone has to be the bigger guy here :)

Can you please make it possible to plug into node's new mjs ESMLoader workflow?

There is a TypeScript issue open already.

Some notes:

• must run node --experimental-modules something.mjs (yes mjs)

  No longer the case, but .mjs is always reserved for ESM.

• that --experimental-modules will go away obviously.

  It did but more --experimental-related flags still linger.

• that kicks in a new layer of loaders in node's internals that completely do not honour conventional require hooks I know you must have hit that already.

  Support for ESM uses a new subsystem that handles all import operations.

• we can't really force it on non-mjs, unless we get the nodejs team is willing to open things a little.

  No longer the case, now package.json supports { "type": "module" }... etc.

• TypeScript already supports "commonjs" as a module, maybe "mjs" too. But it seems they might be recommending npm install renamer -g then renamer -regex --find '\.js^' --replace '.mjs' './outDir/**/*.js' as a solid contribution to making things just work.

  Can't speak to this because I no longer transpile from TS as much.

• not sure how well source maps work yet, but that will be a next step on everyone's agenda.

  Not sure how well sourcemaps are handled by native ESM implementations.

• Some nonstandard ways exist to import/export in .js files (@std/esm and babel) but they are shredded into commonjs on demand.

  Native support for ESM is not as open yet.

• You can transpile inline and produce standard-compliant javascript that does not get shredded into commonjs, as long as you can name it mjs.

  See above!

• You are exceptionally positioned to make a difference here.

  And people want to help if they can (ie know where they would)

Further references:

• Good discussion points on nodejs PR to Allow runMain to be ESM #14369
• ECMAScript Modules in nodejs api docs.
• Related and recently indexed files in nodejs repo.
• ModuleWrap in internal/loader but seems to expose { ModuleWrap } = process.bindings('module_wrap') so not really internal "no-touch" but not officially supported and can break.

[blakeembrey]

@daflair I honestly haven't looked into what supporting ES6 modules natively in node.js will entail yet. Since it doesn't use any of the module loading legacy, I'm not sure how we can hook into it. If you have details, feel free to post them - I'm on vacation one more week and can review at a later date. I'll always accept a PR also 😄

[SMotaal]

I really wish I was suited to PR, but I am a lone developer at the moment. I've played around with ts-node a little and even looked into @std/esm but not babel (that is where I personally drew the line). I struggled to find my way honestly.

Besides, the more I think about it, I realize that this would require a different mechanism than legacy, I would be very interested in more discussion and even collaborating any time.

I guess what I am trying to say is that having a solid understanding of something like ts-node is essential to architect a new mechanism and avoid the hassles of lessons learned.

So please enjoy your vacation, but let's revisit this on your schedule :)

[I'll update notes in this thread in meanwhile]

[demurgos]

Hi,
I just want to add that during the Node discussion about ES modules, loader hooks and preloaded modules were mentioned. The plan is to roll out ES support progressively, but the use case for for custom loaders wasn't forgotten (it's listed as an unsupported feature). The spec mentions module resolution, I guess that Node will also provide a dedicated API eventually. I've just followed the discussions so I don't know when it will be available.

[SMotaal]

It seems that with the exposed (not behind flags except for --experimental-modules) basically mean that you can intercept the first call to Module._load and do whatever you need (like copy or symlink with .mjs) but unfortunately, once you pass off the mjs to the _load, loading is done internally inside an internal Loader that is not accessible.

So I have a few points of discussion on some potential solutions that could be refined with some discussions when possible.

[SMotaal]

There is interesting discussion, and it seems that node is coming with a completely new "ES-module-based hooks" module: resolve and instantiate loader pipeline hooks #15445

From my testing, the existing internal and exposed aspects of the loader seem to reset their prototype chain inside each import's execution context, so a patch was only good for imports that occur directly, but imports by imports did not trigger the hooks as intended (or at least I could not get them to)

So the great news is while I was going all out trying to figure out this intentionally inextensible hell they have been working a opening it up anyways :)

When this lands, it looks like it will make life so much better moving forward with standard modules.

Here is what a custom-loader might look like:
guybedford/node@resolve-hook-rebased

// node/test/fixtures/es-module-loaders/example-loader.mjs
import url from 'url';
import path from 'path';
import process from 'process';

const builtins = new Set(
  Object.keys(process.binding('natives')).filter((str) =>
    /^(?!(?:internal|node|v8)\/)/.test(str))
);
const JS_EXTENSIONS = new Set(['.js', '.mjs']);

export function resolve(specifier, parentModuleURL/*, defaultResolve */) {
  if (builtins.has(specifier)) {
    return {
      url: specifier,
      format: 'builtin'
    };
  }
  if (/^\.{0,2}[/]/.test(specifier) !== true && !specifier.startsWith('file:')) {
    // For node_modules support:
    // return defaultResolve(specifier, parentModuleURL);
    throw new Error(
      `imports must begin with '/', './', or '../'; '${specifier}' does not`);
  }
  const resolved = new url.URL(specifier, parentModuleURL);
  const ext = path.extname(resolved.pathname);
  if (!JS_EXTENSIONS.has(ext)) {
    throw new Error(
      `Cannot load file with non-JavaScript file extension ${ext}.`);
  }
  return {
    url: resolved.href,
    format: 'esm'
  };
}

[SMotaal]

I checked-out guy's branch (which is already 50% approved and CI-checked and I actually got it to work. Of course this is only a custom 9.0.0-pre build of node on my own Mac but I am running node's tests all the same just to see how stable this is for others to actually play around with.

But here is the output:

❯ ./node --experimental-modules --loader ./loader.mjs imports.js

(node:80001) ExperimentalWarning: The ESM module loader is experimental.
exports.js { scope: { this: undefined } }
imports.js { scope:
   { this: undefined,
     'exported from "./exports"': { default: [Object], scope: [Object] } } }

---

Documentation

for reference only

Loader hooks

To customize the default module resolution, loader hooks can optionally be
provided via a --loader ./loader-name.mjs argument to Node.

When hooks are used they only apply to ES module loading and not to any
CommonJS modules loaded.

Resolve hook

The resolve hook returns the resolved module file URL for a given module
specifier being loaded relative to its parent file URL:

import url from 'url';

export async function resolve(specifier, parentModuleURL, defaultResolver) {
  return {
    url: new URL(specifier, parentModuleURL).href,
    format: 'esm'
  };
}

The default NodeJS ES module resolution function is provided as a third
argument to the resolver for easy compatibility workflows.

In addition to returning the resolved file URL value, the resolve hook also
returns a format property specifying the module format of the resolved
module. This can be one of "esm", "cjs", "json", "builtin" or
"addon".

For example a dummy loader to load JavaScript restricted to browser resolution
rules with only JS file extension and Node builtin modules support could
be written:

import url from 'url';
import path from 'path';
import process from 'process';

const builtins = new Set(
  Object.keys(process.binding('natives')).filter((str) =>
    /^(?!(?:internal|node|v8)\/)/.test(str))
);
const JS_EXTENSIONS = new Set(['.js', '.mjs']);

export function resolve(specifier, parentModuleURL/*, defaultResolve */) {
  if (builtins.has(specifier)) {
    return {
      url: specifier,
      format: 'builtin'
    };
  }
  if (/^\.{0,2}[/]/.test(specifier) !== true && !specifier.startsWith('file:')) {
    // For node_modules support:
    // return defaultResolve(specifier, parentModuleURL);
    throw new Error(
      `imports must begin with '/', './', or '../'; '${specifier}' does not`);
  }
  const resolved = new url.URL(specifier, parentModuleURL);
  const ext = path.extname(resolved.pathname);
  if (!JS_EXTENSIONS.has(ext)) {
    throw new Error(
      `Cannot load file with non-JavaScript file extension ${ext}.`);
  }
  return {
    url: resolved.href,
    format: 'esm'
  };
}

With this loader, running:

NODE_OPTIONS='--experimental-modules --loader ./custom-loader.mjs' node x.js

would load the module x.js as an ES module with relative resolution support
(with node_modules loading skipped in this example).

Dynamic instantiate hook

To create a custom dynamic module that doesn't correspond to one of the
existing format interpretations, the dynamicInstantiate hook can be used.
This hook is called only for modules that return format: "dynamic" from
the resolve hook.

export async function dynamicInstantiate(url) {
  return {
    exports: ['customExportName'],
    execute: (exports) => {
      // get and set functions provided for pre-allocated export names
      exports.customExportName.set('value');
    }
  };
}

With the list of module exports provided upfront, the execute function will
then be called at the exact point of module evalutation order for that module
in the import tree.

[bmeck]

Hiya! Let me know if you are looking for any other hooks in particular. The loader is actually much more relaxed that I planned on making it initially to avoid people intentionally spilling internals; but, am open to figuring out hooks for official ways to do use cases.

[SMotaal]

Amazing… :)

I'm kind of exploring quickly some ideas for abstractions on top of that PR to see if there will be any snags.

Cheers

[SMotaal]

Here is a gist (yes, still using gists) of what I have so far…

A simple TypeScript loader using NodeJS 9+ declarative loader API

[SMotaal]

@bmeck I just have to clarify, I'm simply a user of ts-node with a very strong motivation to get to a point where ES2015 modules finally deliver what they really promised. Please don't misunderstand what I've been posting here as the opinion of TypeStrong or the ts-node folks.

That said, I am very happy with the --loader option so far, but how it will pan out when packages are released in the wild, requiring CLI flags in order to actually work without providing mechanisms for this to happen or at least give relevant warnings to consumers automatically is something that can really limit a loader's usefulness.

Some Thoughts

I realize that many loaders will be design-time-relevant, so at this point it is best hidden behind a CLI/bin, but now what happens if more than one loader is to be used?

Some production-relevant loaders can also be important to certain applications, so what happens then, can they be imperatively loaded?

[bmeck]

@daflair

I realize that many loaders will be design-time-relevant, so at this point it is best hidden behind a CLI/bin, but now what happens if more than one loader is to be used?

Per package loaders are being designed, but global loader was first step towards that. You need a concept of per package loaders to be guard against when people are using non-standard behavior, and a mechanism to default to the parent loader (so that APM/code coverage/testing hooks still can be applied).

Some production-relevant loaders can also be important to certain applications, so what happens then, can they be imperatively loaded?

No plans to do so. Is there an example of a loader that is not known ahead of time that you are thinking of? If nothing else, those loaders could always be loaded and we could have a super() like mechanism to get a hold of the loader above them.