Migrate from webpacker to esbuild

We, full stack Rails engineers, have come a long way. We started off as full stack engineers with our backend and frontend all in one framework. We had the asset pipeline (with sprockets opens a new window ) to help us maintain this ecosystem. When the time came we introduced webpacker opens a new window to fill in where sprockets fell short.

In this post we will take a look at how we can take the next step on our journey by migrating from webpacker to esbuild.

• Journey to JS the Rails 5 Way
• jsbundling-rails with esbuild
• Migrating to jsbundling-rails with esbuild
  • Install jsbundling-rails
  • Remove webpack from any development scripts
  • Remove webpacker tags
  • Move entrypoint
  • Remove webpacker
• Some gotchas
  • Optionally add a esbuild.config.js
  • Using jQuery
  • Using the global object
  • Live reloading
• Considerations before migrating to esbuild
• Conclusion
• Further reading and references

Journey to JS the Rails 5 Way

The frontend evolved at a rapid rate and it became hard for browser technologies to keep up. So additional tools such as npm/yarn, webpack and babel arrived on the scene. These tools provided a modern alternative to the asset pipeline for bundling and compiling javascript.

At this point some jumped the fullstack ship and opted for a Rails API + SPA Frontend. The progressive Rails community brought webpacker into this world, which allowed us to remain loyal to the majestic monolith.

Rails 5.2 introduced Webpacker as an alternative javascript compiler. It went on to replace sprockets as the default javascript compiler in Rails 6. So the Rails way was to compile javascript with webpack and leave everything else to the asset pipeline via sprockets.

Webpacker made webpack easy to configure for our Rails applications however it introduced its own set of problems. Fast forward a few years and much has changed since 2017. Alternatives became more attractive due to the improvements in browser technology and friction introduced by Webpack(er).

Rails 7 now provides a new default way to include javascript in our applications, but true to the doctrine opens a new window it allows alternatives for when the default is not fitting. The default way for javascript in Rails 7 makes use of import maps opens a new window .

Import maps is not the right answer for us if we need to transpile or compile our javascript. This means .jsx, .ts and .vue files are not accommodated by import maps (for now). So are we stuck on the Webpacker train indefinitely? Luckily and absolutely not. In the remainder of this article we will discuss how jsbundling-rails opens a new window with esbuild opens a new window could be our next “get off” point.

jsbundling-rails with esbuild

Jsbundling-rails is a gem that provides the necessary configuration that will enable us to make use of the javascript bundler of our choice. Jsbundling-rails simply adds a few rake tasks that creates the entry point and sets the final build path.

In theory we don’t need to stick to any particular javascript bundler. Jsbundling works so long as we maintain the expected entry point and deliver the bundled output to app/assets/builds.

Esbuild is a highly performant javascript bundler. The core version of esbuild does not have all the features that webpacker has. This page opens a new window explains why there are some features esbuild will never support.

For a Rails application with sprinkles of javascript, esbuild core is enough to get the job done.

Migrating to jsbundling-rails with esbuild

Install jsbundling-rails

First we add the gem to our Gemfile.

+ gem 'jsbundling-rails'

Then we bundle install and run the following in the terminal:

./bin/rails javascript:install:esbuild

The installation script provides the default esbuild configuration which includes:

• Updates to the .gitignore file
• A procfile for running multiple processes with foreman
• A app/assets/builds directory
• Updates to the manifest.js
• A app/javascript/application.js entry point file
• A javascript include tag
• A bin/dev script
• Updates to the package.json dependencies

We need to add the build script to our package.json if it was not added by the installation script. So we add the following to the package.json file.

"scripts": { "build": "esbuild app/javascript/*.* --bundle --sourcemap --outdir=app/assets/builds"  }

This default build script will make it possible to compile our js assets by running yarn build.

Remove webpack from any development scripts

We need to remove webpack references from all development scripts, these include Procfile scripts and scripts in our bin directory. The install script we ran earlier added yarn build –watch to our development procfile and this is the only build script required to compile our javascript

Remove webpacker tags

The script we ran in the previous step added a javascript include tag to our application.html.erb file. This tag will include the new javascript entrypoint javascript/application.js for your build script to be included in your application.

However we need to do a project wide search for all instances of javascript_pack_tag and remove these as they were required by webpacker only.

Move entrypoint

We have a new entry point at app/javascript/application.js . We need to move the contents of the webpacker entry point by copying the contents from app/javascript/packs/application.js to app/javascript/application.js.

It is important to convert require statements into import statements with relative paths.

// this
require("controllers")
require("@rails/ujs").start();


// becomes
import "./controllers"
import Rails from “@rails/ujs”
Rails.starts()

We can delete app/javascript/packs once everything has been moved to the app/javascript directory. Paying close attention as we update the relative paths imported into the new application.js file.

Remove webpacker

Remove the following files, but remember to first add any required configuration to your esbuild build script. Take a look at adding an esbuild config file for insight into creating more complex esbuild configurations.

• ./bin/webpack
• ./bin/webpack-dev-server
• ./config/initializers/webpacker.rb
• ./config/webpacker.yml
• ./config/webpack/development.js
• ./config/webpack/environment.js
• ./config/webpack/production.js
• ./config/webpack/test.js

Remove the webpacker gem in our Gemfile and bundle install afterwards.

- gem 'webpacker'

Finally we could also remove the webpacker packages from our package.json.

yarn remove @rails/webpacker webpack-dev-server

Some gotchas

Optionally add a esbuild.config.js

The esbuild API can be accessed from the command line, in Javascript or in Go. The script we added to our package.json shows how to access esbuild from the command line. In some cases it might be more convenient to use Javascript or Go. We will briefly demonstrate how to use Javascript.

Our build script defined earlier in our package.json file included all build options on a single line. For more complex projects an external build script might be preferable.

We can convert the build script we saw earlier. It doesn’t really matter what we call this build script but it might make sense to call it esbuild.config.js. This seems like a naming convention used by other javascript config files.

// esbuild.config.js

require('esbuild').build({
  entryPoints: ['app/javascript/application.js'],
  bundle: true,
  sourcemap: true,
  watch: process.argv.includes("--watch"),
  outdir: 'app/assets/builds',
}).catch(() => process.exit(1))

Notice that we don’t use the app/javascript/*.* entry point. This is because glob expansion is done by our shell and not by esbuild. We would need to include a library such as glob opens a new window to expand the glob pattern first before passing the paths to esbuild.

Now we can update our package.json build script to ”build”: “node esbuild.config.js”. And everything should work just like it did before.

You may want to review the esbuild API opens a new window if you have a more complex webpack configuration that you wish to migrate to esbuild. Keep in mind that some webpack features can only be accomplished with one of the esbuild plugins opens a new window .

Using jQuery

With Webpacker you might have made jQuery available globally by doing something like this:

environment.plugins.prepend(
  'Provide',
  new webpack.ProvidePlugin({
    $: 'jquery/src/jquery',
    jQuery: 'jquery/src/jquery'
  })
)

With esbuild we need to take a different approach to achieve the same result. We cannot simply import jquery like this:

// app/javascript/application.js
import “jquery”
window.jQuery = jquery
window.$ = jquery

The problem is that import statements get hoisted. So any import statement that requires jquery will be hoisted to execute before the window object assignments. This means that imported scripts that depend on jquery will execute before jquery is available.

Instead we create a new file, let us call it jquery.js.

// app/javascript/jquery.js
import jquery from “jquery”
window.jQuery = jquery
window.$ = jquery

And we import this file into our main entry point, application.js like so:

// app/javascript/application.js
import “./jquery”

Now all imports after the jquery import will have access to the jquery window object.

Using the global object

With Webpack we could assign a global variable to the global object. Webpack automatically converted this to the window object opens a new window .

If we need to maintain backward compatibility with the global object in webpack, then we need to add the --define:global=window to our build command. However if backward compatibility is not required then we can very easily search and replace all instances of global with window.

Live reloading

As we have mentioned above esbuild does not support hot module reloading (HMR) opens a new window , it is not even on the roadmap. After being spoiled with HMR via Webpack for a while, this might be a developer luxury we don’t want to go without.

We may not be able to do HMR, but we can add configuration that enables live reloading. Live reloading essentially triggers a page reload whenever a watched file is updated.

For the best developer experience we would need to add Chokidar opens a new window from npm. Chokidar is a file watching library that allows us to watch for changes to any set of files we wish. Combined with the lightning speed of esbuild we can watch and trigger rebuilds with minimal delay.

First we might want to add Chokidar by running the following command in the terminal.

yarn add -D chokidar

Then we would add a file for our esbuild config. As we noted earlier this file’s name is not important. We could call it esbuild.config.js if it makes sense.

#!/usr/bin/env node

const chokidar = require("chokidar");
const esbuild = require("esbuild");
const http = require("http");
const path = require("path");

const clients = [];
const watch = process.argv.includes("--watch");
const watchedDirectories = [
  "./app/javascript/**/*.js",
  "./app/views/**/*.html.erb",
  "./app/assets/stylesheets/*.css",
];
const bannerJs = watch
  ? ' (() => new EventSource("http://localhost:8082").onmessage = () => location.reload())();'
  : "";

const config = {
  entryPoints: ["application.js"],
  bundle: true,
  sourcemap: true,
  incremental: watch,
  outdir: path.join(process.cwd(), "app/assets/builds"),
  absWorkingDir: path.join(process.cwd(), "app/javascript"),
  banner: { js: bannerJs },
};

if (watch) {
  http
    .createServer((req, res) => {
      return clients.push(
        res.writeHead(200, {
          "Content-Type": "text/event-stream",
          "Cache-Control": "no-cache",
          "Access-Control-Allow-Origin": "*",
          Connection: "keep-alive",
        })
      );
    })
    .listen(8082);

  (async () => {
    const result = await esbuild.build(config);
    chokidar.watch(watchedDirectories).on("all", (event, path) => {
      if (path.includes("javascript")) {
        console.log(`rebuilding ${path}`);
        result.rebuild();
      }
      clients.forEach((res) => res.write("data: update\n\n"));
      clients.length = 0;
    });
  })();
} else {
  esbuild.build(config).catch(() => process.exit(1));
}

We don’t want to dive too deeply into the esbuild configuration api, instead we will do a quick overview of this config file.

First we import the required libraries and we set a few variables.

The config contains the build options we pass to esbuild. The heart of this reloading technique is contained in the banner option. Here we specify that we would like to prepend a piece of Javascript to the build artifact. When the --watch flag is passed in, the additional javascript reloads the page whenever it receives a message from localhost:8082.

Next we have a conditional section. We check and if we are watching then we create a local web server using node’s http module. This server can send requests to the browser to trigger a page refresh.

Next we have an asynchronous self-invoking function that defines chokidar’s behavior. It watches for changes in the specified watch directories. It rebuilds when javascript files are updated. And finally it sends a message to the browser.

Now we need to update our build script in our package.json so that it looks like this:

  "scripts": {
    "build": "node esbuild-dev.config.js"
  }

Finally we want to make sure that our Procfile.dev passes in the watch option. The js section of this file must include the --watch option like this.

js: yarn build –-watch

We can now start the development server with ./bin/dev. And to make sure it works we can make a change to any of our javascript, css or erb files. We can watch more files by adding the appropriate directory to the watchDirectories variable.

Considerations before migrating to esbuild

We are not required to move away from webpacker. In fact there is a maintained gem called shakapacker. Internally it still uses the name webpacker so we could easily trial shakapacker to see if it is a good fit.

jsbundling-rails work in conjunction with the assets pipeline (either Sprockets or Propshaft). We need to ensure that either Sprockets or Propshaft is present in our Gemfile.

Be aware that esbuild is in a so-called “late beta” and it has not reached version 1.0.0 yet. This does not mean that a lot of effort is put into maintaining stability and backward compatibility. This is not a deal breaker but keep it in mind for applications where esbuild would not be considered production ready.

Conclusion

There are a few benefits to replacing webpacker with jsbundling with esbuild. For one and likely the most common reason is the sheer build speed of esbuild. For a relatively small application the below chart shows how dramatically slower webpacker is compared to esbuild.

Another reason we might want to migrate to esbuild is because it might be easier to use and maintain. Webpacker is known for its flexibility. Esbuild is less flexible than Webpack by design. This is right up the Rails developer’s alley where convention over configuration has always been a valued characteristic of the tools we use.

The Esbuild’s community is growing and there are many community provided plugins and starter configuration scripts already available. Esbuild is also used in many other developer tools such as Amazon CDK opens a new window and Phoenix opens a new window .

Are you considering upgrading your Rails applications so you can also simplify the javascript part of your application? We are experts at upgrading Rails applications. Reach out so we can help you evaluate the benefits of migrating to jsbundling-rails.

Further reading and references

• Add Live Reload to the esbuild serve issue opens a new window
• BilalBudhani blog post opens a new window
• David Colby blog post opens a new window
• Esbuild Configuration for Javascript Debugging issue opens a new window
• Comparing webpacker to jsbundling-rails opens a new window
• Modern web apps without JavaScript bundling or transpiling opens a new window
• Rails 7 will have three great answers to JavaScript in 2021+ opens a new window
• gorails esbuild videocast opens a new window
• thomasvanholder blog post opens a new window