Support `.mjs` output

[demurgos]

Experimental support for ES modules just landed in Node 8.5 (changelog, PR).
Since ES modules have some parsing and semantic differences, Node decided to use the mjs extension for ES modules (while js is for the "script" target and commonjs modules).

The current Typescript version (2.5.2) supports ES modules emission but uses the js extension by default. It means that to use it with Node, a post-compilation step is required to change the extension from js to mjs. This adds undesirable complexity to use native ES modules support with Node.

A solution would be to add a compiler option to output *.mjs files when emitting ES modules.

Edit (2018-03-22): The propositions below are a bit outdated. I recommend reading the issue to see the progression. See this comment for my current proposition.

Notes:

• It is possible that new parse goals will be added in the next versions of the ES spec. Letting the user provide the extension explicitly (as opposed to a boolean switch) would allow to be forward compatible with other new extension.
• It should still be possible to emit ES modules with the *.js extension, many tools rely on the js extension.
• Another solution would be to have a different extension for the source files: *.mts files would compile to *.mjs, this would be similar to *.tsx and *.jsx.

[kitsonk]

See #10939 (and #9839, #9551, #7926, #7699 and #9670) and this comment.

[demurgos]

@kitsonk
Thanks for posting these link but I am not sure if they are relevant, could you explain them? I found these while checking if the issue was already opened: most of the issues you listed are about input files. This issue is about compiling ts files to mjs instead of js (adding an options to change the compiler output).

[kitsonk]

Basically those say that TypeScript does not want to concern itself with managing extensions like that, as it is already complicated enough. In particular the output extensions run afoul of the TypeScript non-goal of:

4. Provide an end-to-end build pipeline. Instead, make the system extensible so that external tools can use the compiler for more complex build workflows.

If you want .mjs files, it would be best to do something like:

$ npm install renamer -g
$ renamer -regex --find '\.js^' --replace '.mjs' './outDir/**/*.js'

[demurgos]

Thank you very much for clarifying your comment.
I understand that adding any new feature is a burden since it means that it then must be supported for a long time, but I believe that the benefits of the mjs support are worth it: it won't turn tsc in a complex build tool but help TS users.

The long term goal of the Node team is to allow users to author their code using ES modules without paying a cost due to the commonJs modules and script target. This is one of the reasons why the proposals to require "use module" or an export statement (even an empty one) were dropped in favour of using a new extension. See the Node EPS discussions in nodejs/node-eps#57 and nodejs/node-eps#60. If a post-compilation step to rename the files is still required in two years only because .mjs was introduced later then it will just add another burden to remember for years (like the BOM or the different line endings)... Even if there are some discussions to support ES modules with the .js extension ("module" property), the current position is that - if such support is added - it should be used as a fallback for old tooling. .mjs is promoted for new code.

Complex build tools have their place for assets management, dead-code elimination, bundling, minification, etc. These all fall out of scope for TS, but being able to produce runnable code does not. Big projects have their own workflows and use the TS library (either directly or through plugins for other task runners), adding the renaming step is not a big deal for them. On the other hand, a sizeable chunk of projects using Typescript are small/medium sized libraries. These libraries usually only need a single tool: tsc. If they want to support native ES modules, they'll currently have to come up with a command similar to the one posted above by @kitsonk. The problems are that it raises the barrier to entry, causes duplicate effort and makes the build more expensive.

It raises the barrier to entry because newcomers can no longer simply use tsc -p && node index.mjs (--experimental-modules will no longer be required when the use-case for this issue will be relevant) and run their code to try out Typescript. For existing projects, the "build and run" command is usually already defined as an npm script or gulp/grunt/webpack/whatever task, writing this command (or understanding it) will be more difficult: for some persons it will be trivial, for others it'll require a few hours of research. The rate of change in the JS ecosystem is already pretty high, let's try to not worsen it.
Now, a related problem is that it may cause some inconsistencies and duplication: different projects will use different ways to rename their file. I discovered renamer thanks to the message above, but I would have written a gulp task otherwise. Someone else would have used a POSIX-only shell command with broken edge cases or rolled their own helper Node script. To be honest, I think that most good projects would have a good implementation but this lack of standard way to deal with file renames when no build system is already there may increase the "barrier to entry" problem.
Finally, once the manual renaming is configured, it still represents an additional cost. I don't have any measures but for medium projects and using Node to rename the files, it may be longer to start the VM than running the file rename: why not doing it in the same process as TS? Also: what about --watch? How do I make it play well with my custom command? Do you have to have to watch the build directory to rename as the files are emitted?

So far, most of my arguments were about the build complexity: having it done once in TS is better than having each project coming up with its own solution. There is another important reason to have ES modules working out of the box, and I think that it aligns with the goals of TS:

6. Align with current and future ECMAScript proposals.
7. Preserve runtime behavior of all JavaScript code.

ES modules are part of the spec and the TS syntax uses them. These modules have a specified runtime behaviour that cannot be fully replicated in commonJs. It affects among other things circular dependencies, early errors, mutation of exported namespace, etc. Many people won't care, until it bites them. When setting the module option to commonjs, TS does a pretty good job of generating an output with the expected behaviour but it does not trump using real ES modules. So we are in a situation where we can either set an option and have a good approximation using commonJs but if we want the exact spec and runtime behavior we have to jump through hoops and loops.

One last argument is that even if browsers support native ES modules without requiring .mjs, real-world usage of native ES modules will start server-side with Node because you can control its version. Node is important for the whole ecosystem so it would be damageable to just ignore that it uses .mjs for ES modules. You can't just treat it as a "platform detail" when this platform is used by the majority of the TS audience.

To summarize, here are my main points:

• Now that native support exists for ES modules, people will want to use it out-of-the-box.
• Node settled on .mjs and hopes that it'll gain traction. .mjs is Node-specific, but Node is important
• Many libraries only need tsc, requiring a second tool for renaming increases barrier to entry and causes duplicated effort (of varying quality).
• Using a post-compilation command leads to higher build times and does not play well with the watch mode.
• Using native ES modules is the only way to match the spec exactly, TS should not make it hard to use

[SMotaal]

Just an Opinion

As a very strong fan of TypeScript, and obviously a user of node (I guess even TypeScript is) I would really like to see some agreement between them on an issue that has had so much disagreements over many years of rational and sometimes awkward discussions.

I would have loved it if node had a transition phase to make common js (the non-standard) become .cjs over a period of two years.

That said, mjs seems to be the current direction, obviously that non-js file extension (either one) itself does not matter, but the ability for TypeScript to provide a way to control this inline as the originator of the transpiled files is something that really falls to TypeScript.

That said (again), mjs as a locked extension is kind of very opinionated, someone must have realized that forcing js was okay, it was already called js, but forcing mjs, come on, even my spelling checker is nagging me on that one. I hope nodejs will become a little more flexible on that one.

Proposal

Please make it possible to specify mjs as a module format that simply means compiling es2015 modules and calling them mjs :)

[trxcllnt]

It's important to note here that .mjs isn't only about file extensions. running node with the
--experimental-modules flag also means that ES6 module specifiers must be valid urls, so the extension has to be included in the import statement, e.g. import x from './x.mjs

If I'm understanding this right, renaming the files isn't a solution, TS would also need to compile module specifiers with the .mjs extension in the ES2015+/mjs mode. We're keen to support newer versions of node by publishing non-downleveled iterators as es2015+ modules for IxJS, so it would be great to get an answer on this soon.

[dpogue]

TypeScript already allows you to include a .js extension on your imports (even when you're actually importing .ts TypeScript files). The change needed would be to expand that to also allow .mjs.

[trxcllnt]

It's not about the source files including the extension, it's about the compiled files.

If I compile import x from './x' into ES5/CommonJS, TS emits:

var x = require("./x");

If I compile it to ES2015/ESModules with an .mjs extension, TS should emit:

import x from "./x.mjs"

If a library is trying to support both old and new node with CommonJS and ESM side-by-side (as *.js and *.mjs extensions respectively), node only imports the .mjs files if the extension is included in the module specifier.

[SMotaal]

There are some potential updates in the works… including a --loader option that would you to specify a file with a resolve hook that will take the specifier and parent module path, then it could handle extension prioritization as needed. It seems to be very flexible but their goal is to make it declarative and not have people overloading the actual loader functions.

In terms of picking mjs over js, that is only the case in the CJS loader system which uses the now legacy Module prototype with it's hooks, the ones that everyone likes override all the time.

In the current --experimental-modules release, the ESM loader is hard coded to '.mjs' or falls back to the CJS loader.

What I've seen so far over the past few days makes me believe that the ESM loading system will take over with pluggable CJS loading (unless opting to use the legacy-modules by flag or based on the main file).

Check out this PR/discussion

[SMotaal]

@trxcllnt I guess regarding the part about baking the extension right into the import… this applies to import/export everywhere now as part of loader specs.

When ES2015 modules were a spec'd, there were no loader specs, the notion that the extension can change from source file to .js made it a more natural way to go and everyone got too comfortable there. But all loaders (especially web browsers) realized that no-extension means potential security and not to mention load-time drawbacks.

So if TypeScript would continue to work without third-party tooling, it will need to find a strategy to write out standard (not ISO but platform-specific) out-of-the-"compiler"-box projects that will just work in either node or browsers at least depending on the compilerOptions intent specified by the user, and do so without asking the user to mockup some hack to get it to work.

This is just an opinion, but honestly, it feels like it for TypeScript to address.

Checkout this MDN reference but make sure you notice the part where it first said "excluding the .js extension" then in the examples included the extension anyway.

[bmeck]

I'd like to include here that the IANA has an Internet Draft which specifically adds .mjs for COMMON usage that represents the Module goal of ECMAScript . .mjs is not purely a Node.js concern and is even included in an example for browser specs and is supported by a variety MIME DBs already like shared-mime-info.

@daflair can you expand on what

(unless opting to use the legacy-modules by flag or based on the main file).

means?

The next step after that PR is to setup per package loaders to be able to guard against global loader mutation. So, it might be going the direction you think.

When ES2015 modules were a spec'd, there were no loader specs, the notion that the extension can change from source file to .js made it a more natural way to go and everyone got too comfortable there. But all loaders (especially web browsers) realized that no-extension means potential security and not to mention load-time drawbacks.

Interestingly the WHATWG Loader Spec had a very early version in the ECMAScript spec that was removed at the last minute before ES2015! There are interesting other things like the original CommonJS Spec which mandated not to have extensions. We should probably avoid dwelling on the past so much since the different loaders vary so much on these opinions.

Node's EP specced a superset of the WHATWG resolve algorithm that does do various Node idioms like file extension completion. However, the browser has a subset that is safe to use in Node. Still, even with that subset there are problems with dependency trees since "bare" imports are waiting on userland feedback (you can get involved in the hook here), but is mostly left up to intelligent servers and service workers for now.

I'd recommend trying to compile down to the WHATWG compatible specifiers except for bare specifiers for now.

[SMotaal]

@bmeck what I meant by:

unless opting to use the legacy-modules by flag or based on the main file

I was making an assumption that at some point you will have to use something like --disable-es-modules or --legacy-modules which is really an option that simply does the opposite of --experimental-modules (when this flag is the default behaviour) to resort the current loader system and completely bypass anything related to the new loader (ie legacy applications that simply find ways to be incompatible with this existential change to their eco system).

But now that I dug in a little deeper in your recent PR's this might not be your intent…

So it is best to ask you about your intent here 😉?