CommonJS to ESM in Node.js

Yesterday, I refactored Place¹, which is a non-trivial Node.js app, to use ECMAScript Modules (ESM). Here’s a diff of the full set of changes.

This is the approach I took and some of the issues I ran into, in case it helps someone else:

Find and replace

First off, I started by running the following regular expression² to quickly convert CommonJS require syntax to ESM import syntax:

Find

const (.*?)\s*?=\s*?require\(('.*?')\)

Replace

import $1 from $2

While this will do a lot of the work for you, you’ll still have to go in and add .js suffixes manually to any imports that refer to modules using a file path instead of the npm package name.

Next, I updated the export statements:

Find

module.exports = something

Replace

export default something

I did this manually as sometimes that something is an object you were assigning to module.exports and you cannot do that in ESM as what you’re exporting are references (aka live bindings. I’ll come back to this later with a specific example to demonstate what that means.)

Note that .cjs denotes a CommonJS file and .mjs denotes an ESM file in the following examples.

So, an object exported by CommonJS:

// index.cjs
const { name, age, breed } = require('./osky')
console.log(name, age, breed)

// osky.cjs
module.exports = { name: 'Oskar', age: 9, breed: 'Huskamute' }

Becomes the following when using ESM:

// index.mjs
import { name, age, breed } from './osky.mjs'
console.log(name, age, breed)

// osky.mjs
export const name = 'Oskar'
export const age = 9
export const breed = 'Huskamute'

Also, instead of exporting each constant separately in osky.mjs, we could have exported them all in one go:

const name = 'Oskar'
const age = 9
const breed = 'Huskamute'

export { name, age, breed }

These two approaches are functionally equivalent.

You could also be tempted to keep the default export and destructure it after import:

// index.mjs
import osky from './osky.mjs'
const { name, age, breed } = osky
console.log(name, age, breed)

// osky.mjs
export default { name: 'Oskar', age: 9, breed: 'Huskamute' }

While this will result in identical behaviour in this example, that’s because the live bindings we’re exporting are constants. Let’s see what would happen if we were exporting variables whose values could be changed after the import by the module we’re importing:

// index.mjs
import { name, age, breed } from './osky.mjs'

function updateStatus () {
  console.log(`${name} (a ${breed}) is now ${age} year${age > 1 ? 's': ''} old.`)
}

updateStatus()
setInterval(updateStatus, 1000)

// osky.mjs
export let name = 'Oskar'
export let age = 1
export let breed = 'Huskamute'

setInterval(() => age++, 1000)

When you run this, you will see that Oskar’s age increases by one every second and is reflected in the status update even though the updates take place in the module being imported and are logged in index.mjs. This is what we mean when we say ESM modules export live bindings. Basically, what you’re exporting is a reference to your original variable.

Now, change the export to the second approach we used:

// index.mjs
import osky from './osky.js'
const { name, age, breed } = osky
// …

// osky.mjs
let name = 'Oskar'
let age = 1
let breed = 'Huskamute'

setInterval(() => age++, 1000)

export default { name, age, breed }

When you run this version, you’ll notice that Oskar never ages. That’s because when we destructure, we no longer have a reference to the originally-imported variable and the updates taking place in the original module are not reflected in the console output.

If you did want to use a default export and keep the live bindings, this is how you’d do it:

// index.mjs
import dog from './osky.js'

function updateStatus () {
  console.log(`${dog.name} (a ${dog.breed}) is now ${dog.age} year${dog.age > 1 ? 's': ''} old.`)
}

updateStatus()
setInterval(updateStatus, 1000)

// osky.mjs
const name = 'Oskar'
const age = 1
const breed = 'Huskamute'

const osky = { name, age, breed }

setInterval(() => osky.age++, 1000)

export default osky

Note that I’ve purposefully stored the original values, as well as the osky object in constants in osky.mjs to illustrate the point even further of what’s being exported and what’s changing. (Remember that marking an object as a constant means that you cannot assign a different object to it, not that you cannot change the values of its properties or even add or remove properties from it.)

Anyway, all this to say that you should keep in mind that ESM modules are not just a different syntax, they are fundamentally different to CommonJS modules in how they work.

So, with the find and replace out of the way, I had to tackle a few other issues.

Dynamic requires and imports

In Place, I was using dynamic requires to load in routes for the server. I needed to change these to use dynamic imports instead so you could use ESM for the server as well as the client.³

This is not as straightforward as a simple find and replace because while dynamic requires are synchronous, dynamic imports are asychronous. And asynchronous code has a cascading effect⁴.

(Note that you can still do an old-school, synchronous require. You just have to create your require() function yourself. See A require by any other name…, below, for details.)

The asynchronous cascade

For example, refactoring function c(), below, to return a promise means that functions b() and a() become asynchronous also:

// Sync
function a () { console.log(b()) }
function b () { return c() }
function c () { return 'Hello' }

a()

// Async
async function a () { console.log(await b()) }
async function b () { await c() }
function c () {
  return new Promise (resolve => setTimeout(resolve, 3000)
}

await a()

Awaiting imports

The switch from synchronous dynamic requires to asynchronous dynamic imports was the biggest change in the refactor. Here’s a relevant section of the diff, showing the before and after:

-decache(routesJSFilePath)
-require(routesJSFilePath)(this.app)
+// Ensure we are loading a fresh copy in case it has changed.
+const cacheBustingRoutesJSFilePath = `${routesJSFilePath}?update=${Date.now()}`
+;(await import(cacheBustingRoutesJSFilePath)).default(this.app)

So requiring a default function from a module and immediately executing it goes from:

require(filePath)(args)

to

;(await import(filePath)).default(args)

Note the addition of .default. That’s how you access the default export of a dynamically-imported module.

Also note where the parentheses are around await. The promise must resolve before you can access the module’s properties.

And here’s something to watch out for: if you forget to await an asynchronous call when using dynamic imports, Node.js will throw a very unhelpful error message without a stack trace:

SyntaxError: Unexpected reserved word

If you encounter that error, chances are you should be checking for missing await statements in your code.

Cache busting with dynamic ESM imports

Note: This will leak memory and eventually crash your system. Garbage collecting stale ESM modules is not a solved problem as of Feb, 2021.

In the example above, you can see that I also had to change the cache-busting strategy for when routes are reloaded at runtime.

With dynamic requires, I was using the decache module.

With dynamic imports, you don’t have programmatic access to the module cache so I’m using an old client-side trick instead and appending a query string to the file path with the current timestamp in it.

If you want to test this out quickly for yourself to see that it works, create a file called loader.mjs with the following code in it:

setInterval(async () => {
  const cacheBustingPath = `./message.mjs?update=${Date.now()}`
  console.log((await import(cacheBustingPath)).default)
}, 1000)

Then, create another file called message.mjs that contains the following line of code:

export default 'hello'

Finally, run the loader:

node loader.mjs

You should see the message hello printed out to your console every second.

With that process still running, change the message in message.mjs and save the file and you will see the message update in the console.

Will no one think of the loops?

When using dynamic imports and asychronous calls in linear tasks, remember that you can’t just loop over promises like you can with synchronous calls, you must await each result.

So no forEach(), etc., for you.

Here’s a relevant diff from where I changed a synchronous forEach() loop into an asychronous loop using Sebastien Chopin’s handy asyncForEach() function:

-httpsPostRoutes.forEach(route => {
+asyncForEach(httpsPostRoutes, async route => {
-  this.app.post(route.path, require(route.callback))
+  this.app.post(route.path, (await import(route.callback)).default)
 })

And this is the asyncForEach() function itself:

async function asyncForEach(array, callback) {
  for (let index = 0; index < array.length; index++) {
    await callback(array[index], index, array);
  }
}

__dirname

Another issue that cropped up was the lack of __dirname support with ESM under Node.js. This is easily fixed by declaring __dirname as a global in your module:

import { fileURLToPath } from 'url'
const __dirname = fileURLToPath(new URL('.', import.meta.url))

This is such an easy, backwards-compatible fix that I don’t know why Node.js doesn’t automatically inject it into all ESM modules.⁵

Note: This snippet was previously using the .pathname property of the URL instance, which is wrong and won’t work on Windows. The updated snippet, above, properly converts the URL to a file path.

A require by any other name…

A problem with a similar solution to the missing __dirname functionality exists if you have code that uses require to load in JSON files. ESM modules do not currently support this. So what can you use, given you can’t use require with ESM modules?

Why, require, of course…

No, that’s not a typo.

You just have to create your own require function.

(I know, I know… I’m shaking my head too.)

import { createRequire } from 'module'
const require = createRequire(import.meta.url)

Then, you can require a JSON file just like you did before. e.g., along with the __dirname constant, above, the following code will work if you have a file called my.json in the same directory as your script:

JSON.parse(fs.readFileSync(path.join(__dirname, 'my.json'), 'utf-8')))

Again, unless there are some edge cases that the Node development team knows of that I don’t, I don’t see why this isn’t default behaviour that’s injected into ECMAScript Modules in Node.js.

The above two features – alongside doing the same for __filename, etc. – would go a long way to making the migration process from CommonJS to ESM a far more seamless one for many developers.

Little quirks

The final thing I had to do was to set the type property to module in my package.json. When you do this, all .js files are treated as ESM by Node.js. Which can cause some unexpected quirks.

For example, JSDB – my small, in-memory, streaming write-on-update JavaScript database – persists to JavaScript transaction logs and uses a CommonJS require to load them in.⁶ Even though I was importing JSDB in from node_modules, it crashed while trying to load data tables:

Error [ERR_REQUIRE_ESM]: Must use import to load ES Module: /home/aral/small-tech/small-web/sandbox/sign-in.small-web.s
require() of ES modules is not supported.
require() of /home/aral/small-tech/small-web/sandbox/sign-in.small-web.org/.db/privateRoutes.js from /home/aral/small-.
Instead rename privateRoutes.js to end in .cjs, change the requiring code to use import(), or remove "type": "module" .

    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1080:13)
    at Module.load (internal/modules/cjs/loader.js:928:32)
    at Function.Module._load (internal/modules/cjs/loader.js:769:14)
    at Module.require (internal/modules/cjs/loader.js:952:19)
    at require (internal/modules/cjs/helpers.js:88:18)
    at JSTable.load (/home/aral/small-tech/small-web/place/app/node_modules/@small-tech/jsdb/lib/JSTable.js:161:20)
    …

There seems to be some context leak between an app and the npm modules it imports whereby the type of the app overrides the type of the npm module declared in the module’s package file when using a dynamic require().

The quick workaround I hacked together for the moment was to add a default (of type CommonJS) package.json file to the database directory. In fact, I might just update JSDB to do that by default so that it can be used in both CommonJS and ESM projects with a minimum of fuss.

It feels hacky but I don’t think it’s worth the effort to create a non-hacky solution to something that is itself a huge hack (ESM in Node.js).

The same issue also cropped up when I used Snowpack’s loadConfiguration() method (which uses require() internally) to load the Snowpack configuration for Place. The easy workaround to that was just to rename snowpack.config.js to snowpack.config.cjs. The irony of having to make my Snowpack configuration file a CommonJS file when Snowpack exists to make ESM easy to use is not lost on me.

Other quirks

Error stacks are different between CommonJS and ESM

This is one you will probably not care about but it just bit me while converting Auto Encrypt to ESM as I have custom error handling that uses the information provided in stack traces.

To see the difference for yourself, create two files, index.cjs and index.mjs, both with the following line of code in them:

console.log((new Error().stack))

The output you’ll see in each will differ (the actuals paths you see will, of course, be based on the directory structure of your local system).

CommonJS:

Error
    at Object.<anonymous> (/home/aral/sandbox/stack-get-file-name-esm-vs-commonjs/index.cjs:1:14)
    at Module._compile (internal/modules/cjs/loader.js:1063:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1092:10)
    at Module.load (internal/modules/cjs/loader.js:928:32)
    at Function.Module._load (internal/modules/cjs/loader.js:769:14)
    at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:72:12)
    at internal/main/run_main_module.js:17:47

ESM:

Error
    at file:///home/aral/sandbox/stack-get-file-name-esm-vs-commonjs/index.mjs:1:14
    at ModuleJob.run (internal/modules/esm/module_job.js:152:23)
    at async Loader.import (internal/modules/esm/loader.js:166:24)
    at async Object.loadESM (internal/process/esm_loader.js:68:5)

Just be aware of this if you are doing anything based on the contents of the error stack, especially with paths as ESM returns URLs whereas CommonJS returns file paths.

Conclusion

Prior to native support for ECMAScript Modules in Node.js, I saw no reason to use them. Now, I take it as given for future Node projects and look forward to working more easily with isomorphic JavaScript.

I hope this post with my experiences from yesterday’s refactor are helpful for you if and when you undertake a similar task.