What does a JavaScript Transpiler do?

Rationale

JavaScript compilers, called transpilers, allow development to occur across multiple files without impacting network performance.  Transpilers help to optimize large projects worked on by multiple developers (coming soon: developing projects that scale).  On the web, serving files incurs latency costs per request (HTTP 2.0 might help), and worse, can delay the initialization of a page while a resource remains unloaded.  Transpilers solve this problem by combining the necessary JavaScript resources into one file called a bundle.  Servers can send one bundle instead of many files, even if a JavaScript application exists across dozens of files.

Transpilers create cross-compatability across JavaScript versions and translate pseudo-langauges like CoffeeScript and TypeScript into JavaScript.  Further, transpilers can perform the additional step of minifying files to further reduce filesize and therefore network latency.  To summarize the basic usage of transpilers:

  • Condense many development files into one file called a bundle
  • Translate superset languages into JavaScript and translate newer versions of JavaScript into older versions of JavaScript
  • Perform additional build steps not needed for development such as file compression

Why Develop Across Multiple Files?

Files remain the base unit for organizing the development of applications.  Spreading development across multiple files allows a developer to leverage the filesystem’s organization and structure to bring structure to the development process.  Importantly, this allows multiple developers to work on multiple files at once since tools for publishing code typically make updating separate files easy.

Are Transpilers Used only for Browser-Based JavaScript Applications?

The primary rationale of transpilers is to optimize the deployment of JavaScript to the browser.  Node.js applications can be transpiled as well.  It’s easier to distribute one executable to a user is easier than requiring that user to have a Node.js environment installed.  Transpilers help Node.js applications distributing closed-source applications which assist freemium business models.  Transpiling Node.js code may also ease deployment to less accessible device environments.  Here are three Node.js compilers available:

What Problems Must a Transpiler Solve?

Transpilers fundamentally create bundles.  They take multiple source files and combine them into a single file called a bundle or in Node.js an executable.  Transpilers must determine what to bundle.   When bundling, transpilers must virtualize import and require calls so that references to other bundled files are resolved internally within the bundle.

In addition to the core bundling problem, transpilers must translate newer versions of JavaScript or other languages into portable JavaScript.  Finally, transpilers compress files and optimize their delivery over a network.

Determining What to Bundle?  Entry-Point vs. Glob

Two basic methods exist to determine what files are included in a bundle.  The entry-point method of file bundling utilizes a main file and overrides import and require calls to determine other files to bundle.  The advantage of the entry-point method is that it’s simple.  The entry-point method requires little to no developer oversight and let’s the transpiler optimize the bundle using the mininum number of files.

File globs operate on Unix-style pattern-matching to include files in the bundle.  File globs require precise control from the developer, but allow developers to assert control over which bundles contain which files.  It’s ideal to hand over control to a transpiler to perform optimizations, but every application is unique.  In addition, the technologies used to build JavaScript applications change frequently.  As a result, optimizing JavaScript can be subjective and delegating optimization to a transpiler not perfectly efficient.

The concepts of entry-point and file globs are not new.  Java and C++ by default must define entry-point functions called main.  Likewise, JavaScript transpilers use file-based entry-points. Given an entry-point, a compiler or transpiler is able to evaluate files and resources imported and required by the entry-point.  Using the list of imported and required files, a transpiler automatically generates a list of files to be contained in the bundle.  In traditional compilers, this optimization is typically performed in a linker.

Resolving Require and Import Calls to Bundled Files

After a list of files to bundle are determined, the transpiler must consolidate the source files into one file called a bundle.  This presents a minimal set of challenges:

  • Files must be included in an order that they are depended on by other internal modules, i.e. file contents should be in order AND OR
  • File contents should not overwrite variables and functions of other modules, i.e. file contents should be scoped
  • File contents of internal modules and external modules must resolve require and import statements of internal bundle modules to the bundle file, not the originating file location, i.e. module references should be resolved

Let’s have a look at webpack’s require function override:

Explaining webpack’s Boilerplate

By default, webpack bundles a function where the the entry-point is executed by default, and all require/import statements are overridden by an internal __webpack_require__ function.  This can be seen on line 33:

This runs the module[0] code and includes the __webpack_require__ function in place of the default Node.js require function. Notably, on line 20 the internal webpack modules are required via the __webpack_require__ function:

This line ensures that all require statements internal to the bundle are called with __webpack_require__ instead of the native Node.js require statement. This allows webpack to transform Node.js require statements into a browser compatible format.

RegExp vs. Function Splicing

Since RegExp is generally slow, it’s important to note that RegExp is not required for solving most bundling challenges.  Overriding the require function is sufficient.  However, for ES6 modules using import statements, it’s impossible to use the function splicing technique.  This is further complicated because  import statements must be included at global/file scope.  In general, slowness in bundling is usually attributed to the complexity of RegExp calls on file resources.

Deploying Bundles

Given that a library has a repeatable way to build a list of included files in a bundle, it is ready to create a bundle.  When a library can solve the bundle challenges of order, scope and references, a bundle can be used in place of loading multiple development files.  Two primary costs must be evaluated when considering when bundling:

  • How long does a bundle take to build?
  • What is the resulting size of the bundle, and how long does it take to serve over a network?

Both of these metrics can be broken down further.  When considering build times, it’s possible to consider initial build time vs. incremental build time.  Incremental builds always use the delta or changes between builds to update the bundle.  There is always a tradeoff between detecting incremental changes, and the computation cost of editing only part of a bundle.

Likewise, steps like minification can help reduce the size of a bundle, but breaking the bundle into chunks can speed delivery.  Given that a bundle provides the resources needed to load a page, bundles can be segmented to deliver the resources needed to load only a portion of the application experience.  Typically, when segmenting bundles, a bundle should be created to serve the common set of resources needed across the whole application.

Development Builds: Clientside require vs. Bundling Everything

Assuming that a transpiler is bundling for the browser, it’s worthwhile to consider whether a bundle is required for the browser to load the application.  Systems like webpack require a bundle to be built even for trivial changes.  This can dramatically impact development time.  However, it’s possible to leverage a browser to treat require statements as XHR network requests and wait for the dependencies to finish loading before initializing a resource.  This relies on a local connection reducing network response times to a couple milliseconds.

Rollup.js Tree-Shaking

Tree-shaking eliminates “dead code” by utilizing “static analysis”, according to Rollup’s website.  The website’s documentation indicates leveraging ES6 modules to do so.   Rollup is analyzing import statements and only tracking the functions and variables imported for bundling.  This is a tradeoff of computation time to use RegExp to analyze import statements and the decreased bundle size resulting from the optimization.

Determining What Version of JavaScript to Run

Coming soon…

Conclusions

Transpilers often brand themselves as fully automated, but require configuration.  Mature environments in Java and C++ require little configuration since the underlying language and output expectations change little.  In JavaScript, this is not the case, and transpilers are continuously adapting to changes in standards and development expectations.

RegExp is computationally costly, and transpiling JavaScript for backwards-compatibility is consequently slow.

Many bundling libraries do not provide access to development builds without the bundling step.  When bundling also requires transpiling JavaScript for backwards-compatability, this can impact the efficacy of a development team.

Segmenting between speedy development builds and efficient production builds is often left as unclear for developers.  Most developers are not used to deeply configuring deployments, and many DevOps skillsets are not likewise involved enough in JavaScript technologies to make an impact.  It’s important for architect stakeholders to understand these challenges before adopting JavaScript transpilers.

Further Reading

Leave a Reply

Your email address will not be published. Required fields are marked *