What does a JavaScript Transpiler do?

Rationale

JavaScript compilers, called transpilers, allow development to occur across multiple files without impacting network performance.  Transpilers help to optimize large projects worked on by multiple developers (coming soon: developing projects that scale).  On the web, serving files incurs latency costs per request (HTTP 2.0 might help), and worse, can delay the initialization of a page while a resource remains unloaded.  Transpilers solve this problem by combining the necessary JavaScript resources into one file called a bundle.  Servers can send one bundle instead of many files, even if a JavaScript application exists across dozens of files.

Transpilers create cross-compatability across JavaScript versions and translate pseudo-langauges like CoffeeScript and TypeScript into JavaScript.  Further, transpilers can perform the additional step of minifying files to further reduce filesize and therefore network latency.  To summarize the basic usage of transpilers:

  • Condense many development files into one file called a bundle
  • Translate superset languages into JavaScript and translate newer versions of JavaScript into older versions of JavaScript
  • Perform additional build steps not needed for development such as file compression

Why Develop Across Multiple Files?

Files remain the base unit for organizing the development of applications.  Spreading development across multiple files allows a developer to leverage the filesystem’s organization and structure to bring structure to the development process.  Importantly, this allows multiple developers to work on multiple files at once since tools for publishing code typically make updating separate files easy.

Are Transpilers Used only for Browser-Based JavaScript Applications?

The primary rationale of transpilers is to optimize the deployment of JavaScript to the browser.  Node.js applications can be transpiled as well.  It’s easier to distribute one executable to a user is easier than requiring that user to have a Node.js environment installed.  Transpilers help Node.js applications distributing closed-source applications which assist freemium business models.  Transpiling Node.js code may also ease deployment to less accessible device environments.  Here are three Node.js compilers available:

What Problems Must a Transpiler Solve?

Transpilers fundamentally create bundles.  They take multiple source files and combine them into a single file called a bundle or in Node.js an executable.  Transpilers must determine what to bundle.   When bundling, transpilers must virtualize import and require calls so that references to other bundled files are resolved internally within the bundle.

In addition to the core bundling problem, transpilers must translate newer versions of JavaScript or other languages into portable JavaScript.  Finally, transpilers compress files and optimize their delivery over a network.

Determining What to Bundle?  Entry-Point vs. Glob

Two basic methods exist to determine what files are included in a bundle.  The entry-point method of file bundling utilizes a main file and overrides import and require calls to determine other files to bundle.  The advantage of the entry-point method is that it’s simple.  The entry-point method requires little to no developer oversight and let’s the transpiler optimize the bundle using the mininum number of files.

File globs operate on Unix-style pattern-matching to include files in the bundle.  File globs require precise control from the developer, but allow developers to assert control over which bundles contain which files.  It’s ideal to hand over control to a transpiler to perform optimizations, but every application is unique.  In addition, the technologies used to build JavaScript applications change frequently.  As a result, optimizing JavaScript can be subjective and delegating optimization to a transpiler not perfectly efficient.

The concepts of entry-point and file globs are not new.  Java and C++ by default must define entry-point functions called main.  Likewise, JavaScript transpilers use file-based entry-points. Given an entry-point, a compiler or transpiler is able to evaluate files and resources imported and required by the entry-point.  Using the list of imported and required files, a transpiler automatically generates a list of files to be contained in the bundle.  In traditional compilers, this optimization is typically performed in a linker.

Resolving Require and Import Calls to Bundled Files

After a list of files to bundle are determined, the transpiler must consolidate the source files into one file called a bundle.  This presents a minimal set of challenges:

  • Files must be included in an order that they are depended on by other internal modules, i.e. file contents should be in order AND OR
  • File contents should not overwrite variables and functions of other modules, i.e. file contents should be scoped
  • File contents of internal modules and external modules must resolve require and import statements of internal bundle modules to the bundle file, not the originating file location, i.e. module references should be resolved

Let’s have a look at webpack’s require function override:

Explaining webpack’s Boilerplate

By default, webpack bundles a function where the the entry-point is executed by default, and all require/import statements are overridden by an internal __webpack_require__ function.  This can be seen on line 33:

This runs the module[0] code and includes the __webpack_require__ function in place of the default Node.js require function. Notably, on line 20 the internal webpack modules are required via the __webpack_require__ function:

This line ensures that all require statements internal to the bundle are called with __webpack_require__ instead of the native Node.js require statement. This allows webpack to transform Node.js require statements into a browser compatible format.

RegExp vs. Function Splicing

Since RegExp is generally slow, it’s important to note that RegExp is not required for solving most bundling challenges.  Overriding the require function is sufficient.  However, for ES6 modules using import statements, it’s impossible to use the function splicing technique.  This is further complicated because  import statements must be included at global/file scope.  In general, slowness in bundling is usually attributed to the complexity of RegExp calls on file resources.

Deploying Bundles

Given that a library has a repeatable way to build a list of included files in a bundle, it is ready to create a bundle.  When a library can solve the bundle challenges of order, scope and references, a bundle can be used in place of loading multiple development files.  Two primary costs must be evaluated when considering when bundling:

  • How long does a bundle take to build?
  • What is the resulting size of the bundle, and how long does it take to serve over a network?

Both of these metrics can be broken down further.  When considering build times, it’s possible to consider initial build time vs. incremental build time.  Incremental builds always use the delta or changes between builds to update the bundle.  There is always a tradeoff between detecting incremental changes, and the computation cost of editing only part of a bundle.

Likewise, steps like minification can help reduce the size of a bundle, but breaking the bundle into chunks can speed delivery.  Given that a bundle provides the resources needed to load a page, bundles can be segmented to deliver the resources needed to load only a portion of the application experience.  Typically, when segmenting bundles, a bundle should be created to serve the common set of resources needed across the whole application.

Development Builds: Clientside require vs. Bundling Everything

Assuming that a transpiler is bundling for the browser, it’s worthwhile to consider whether a bundle is required for the browser to load the application.  Systems like webpack require a bundle to be built even for trivial changes.  This can dramatically impact development time.  However, it’s possible to leverage a browser to treat require statements as XHR network requests and wait for the dependencies to finish loading before initializing a resource.  This relies on a local connection reducing network response times to a couple milliseconds.

Rollup.js Tree-Shaking

Tree-shaking eliminates “dead code” by utilizing “static analysis”, according to Rollup’s website.  The website’s documentation indicates leveraging ES6 modules to do so.   Rollup is analyzing import statements and only tracking the functions and variables imported for bundling.  This is a tradeoff of computation time to use RegExp to analyze import statements and the decreased bundle size resulting from the optimization.

Determining What Version of JavaScript to Run

Coming soon…

Conclusions

Transpilers often brand themselves as fully automated, but require configuration.  Mature environments in Java and C++ require little configuration since the underlying language and output expectations change little.  In JavaScript, this is not the case, and transpilers are continuously adapting to changes in standards and development expectations.

RegExp is computationally costly, and transpiling JavaScript for backwards-compatibility is consequently slow.

Many bundling libraries do not provide access to development builds without the bundling step.  When bundling also requires transpiling JavaScript for backwards-compatability, this can impact the efficacy of a development team.

Segmenting between speedy development builds and efficient production builds is often left as unclear for developers.  Most developers are not used to deeply configuring deployments, and many DevOps skillsets are not likewise involved enough in JavaScript technologies to make an impact.  It’s important for architect stakeholders to understand these challenges before adopting JavaScript transpilers.

Further Reading

What is a Hybrid Cryptosystem in Node.js? (in 5 steps)

Introduction

Securing communication over the internet using RSA is computationally costly.  If your server needs to handle thousands of concurrent users to validate multiple requests per session, it becomes impractical to keep response times reasonable.

Symmetric encryption is less computationally costly, but creates the challenge of synchronizing the secret between client and server.  It’s therefore possible to utilize RSA to securely communicate a symmetric key shared between each unique client and server.

  1. A public/private keypair is generated by the client.  In Node.js, it’s most practical to use the OpenSSL executable bundled with the installation.  The public key is sent to the server to request a new session.
  2. The session is created as a GUID (globally unique identifier) and encrypted with the transmitted public key.  Since RSA is a two-step encryption/decryption, the public key can be known publicly by eavesdroppers without comprising the privacy of communications.
  3. The publicly encrypted session secret is then sent back to the client and decrypted with the private key.  The session secret is now known between both server and client.
  4. Using a symmetric encryption algorithm like AES-256-GCM, sensitive data can be secured with the session secret and sent across an eavesdroppable network.
  5. The server receives the encrypted data and is able to decrypt it using the session secret.  If need be, this process can be reversed where the server encrypts data using the session secret and the client could decrypt it.

Differentiating Asymmetric/Symmetric/RSA Cryptography

Understanding the differences between asymmetric/symmetric/RSA encryption is trivial enough. An asymmetric cryptogram is a one-way hash. Once the value is converted, it cannot be undone. You typically use this for passwords, such that if the accounts database is compromised, the passwords cannot be read in plaintext.  A salt is an arbitrary and usually random string added to each password such that if all accounts are compromised then a brute force attack’s work cannot be reused across accounts.

A symmetric cryptogram is typically what we think of with cryptography.  A secret key encrypts a message that can safely be sent publicly because it’s without the key the message cannot be read.

Symmetric cryptography becomes problematic when communicating the secret key must be done with a computer over an insecure network.  RSA is a category of algorithm that generates a two-step encryption.  In other words, the pattern encrypt(message, key1) and decrypt(encrypt(message, key1), key2) must be followed to decrypt a message.  With this mathematical relationship, key1 can be shared over a network without compromising secrecy.  Likewise, since encrypt(message, key1) is only dicipherable with key2, the encrypted message is safe to send over the network.

A hybrid cryptosystem sounds fancy, but it’s simple.  RSA is used to communicate a secret key, and every subsequent communication utilizes symmetric encryption.

 

Asymmetric Symmetric Public-Key  Cryptographic Systems
Example Algorithm SHA-512 AES-256-CTR RSA
Example Input somePassword Secret message in a bottle theKeyToMy<3
Processed Input somePassword_0d282a18-9a6e-11e7-abc4-cec278b6b50a Secret message in a bottle83ba65c977849f45 21e32b85abab6d8f1e42d91b0319feecd23f555e20ac7693cae7dac779de60be
Key1 theKeyToMy<3 /usr/home/public.pem
Key2 /usr/home/private.pem
Example Output c9b175ebc7dff3dcd5a40ceeaef92491f1b8e134b81216313fba8791a0f30cee MDAyMjQ4Y2NhZmExODY4OA==/1C3cbYSQ771FK5jakvPXG44BTL/dvjwxfk= theKeyToMy<3
Typical Usage Password Storage Sharing secret data Key exchanges

Nonce’s, Timestamps and Sharding

At least three other concepts are important for cryptographic systems.  Encrypted messages created with an initialization vector create a similar impact to salts in an asymmetric cryptosystem; they become harder to brute force.  Furthermore, a HMAC (hash-based authentication code) can be generated to verify integrity of symmetrically encrypted message.  HMAC’s are similar to CRC’s except that the HMAC requires a HMAC to be generated by both parties using the shared and secret key, whereas a CRC is calculated by a message only.

While the HMAC can be brute-forced such that a message be modified and such that two HMAC’s match regardless of the shared secret, this is computationally costly.  By utilizing nonce’s and enforcing timed expiration on messages, it becomes computationally impossible to 1) brute-force a message within the expiration time (such that a forged request cannot be submitted hours later) 2) build a lookup table to increase tampering efficiency over time (since every message is uniquely impacted by its nonce).

Finally, sharding techniques include distributing authentication mechanisms across machines.  In order for authentication to occur, multiple machines must verify a credential.  Two-factor authentication is an easy of example of this.  Both a phone and a computer must be used to verify a credential.  This technique relies on differing security models for each system and the number of systems used.  Compromising a sharded authentication system requires virtualizing more devices (e.g. a cell phone OS and a desktop OS) and this exponentially increases the costs of exploitation since virtualization always is subjected to the underlying system changing and running a cluster of computers is generally more difficult than operating a single process/instance.

Further Reading

A StackOverflow expert explains the importance of salts and provides a glimpse into the mathematical complexity of describing cryptographic systems.

Blockchain utilizes asymmetric encryption to create a “block” which is the unit of currency for Bitcoin.  The theory is that it’s computationally many times easier to verify a set of asymmetric hashes than it is to generate an asymmetric hash that satisfies the sequencing requirement of blockchain.

Finally, Google recently proposed distributed machine learning where users belonging to a cluster of computers could be certain of guaranteed privacy across a set of machines.  They coined the term for this combination of encryption/machine learning computation/and sharding as “federated learning”.

How to get a CSS Selector given a HTML Document?

Abstract

It’s relevant when discussing development tools to obtain a CSS Selector for HTML documents.  Chrome DevTools’ Inspect Feature allows for this using a point-and-click interaction.  This interaction is useful for web-based editors like WebFlow and for web scrapers like ParseHub.  It’s additionally relevant when using stringification methods to update virtual DOM trees in JavaScript Frameworks.  Iterative methods are quicker for reverse engineering a CSS selector given that an HTMLElement has exactly one parent.  However, due to the stringification requirement of efficiently building virtual DOM trees, I’ll cover the stringification method generating an :nth-child selector.

Selection Strategies

In a web scraper or website builder, it’s helpful to consider selectors matching multiple elements of the same kind.  Commonly, this is done using CSS class/ID selections and common ancestors in the DOM tree.  While not covered in this post, it’s important to heuristically refine a selection to match the clicked elements by a user even when elements are contained in disparate trees.  If no class/ID selections are used, then it’s important to rely on the :nth-child pseudo-class.

Stringification methods are additionally useful when considering the broader topic of reducing O(n log(n)) search times in a tree to O(n).  This rationale exists because DOM trees are unsorted and string contents must be searched on each iteration.  The same methodology applies to reconstructing JSON objects.

Stringification Requirements

To build a CSS selector, first a set of regular expressions must be generated to match an HTML document:

  • Open Elements (e.g. <p>)
  • Closed Elements (e.g. </strong>)
  • All elements (e.g. <p> and </strong>)
  • Self-Closing Elements (e.g. <img />)

The example function utilizes these regular expressions:

The function reads back all nodes from the offset.  This may look like the following chart:

Put into a table, we can determine what the path of the left-most tag may look like:

tag name index type include? path
p 0 open yes p
footer 1 open yes footer > p
p 2 close no footer > p
strong 3 close no footer > p
strong 4 open no footer > p
img 5 self-closing no footer > p
p 6 open no footer > p
body 7 open yes body > footer > p

Arranged as a tree, it becomes easier to see the requirements to obtain a correct path more clearly:



Note: the above tree is not visually sorted in left to right order to determine rank.

The code might start to look like this:

Basic HTML Heuristics

To reverse engineer a CSS path via an HTML string, it becomes important to define general rules about the path of a node:

  • Only open tags can be candidates for path inclusion
    • Self-closing tags per this rule are always excluded
  • All close tags must have their matching open tag met before an open tag may be considered for path inclusion

The first heuristic is most important.  This implies that we need to call parts of the array beginning with a closed element until that closed element is opened.  This is non-trivial when we consider a tree may contain multiple elements of the same name, i.e. searching for the first opened element of the same name may be incorrect.  Since in XML and HTML each close tag must have a matching open tag and because each tag has exactly one parent, then it becomes possible to consider the innermost closed tag and work outwards to find matching tags for previously discovered close tags.  A stack is the appropriate data structure to elegantly perform this operation.

To cull an irrelevant subtree, the array must be iterated through starting at the index after the first close tag.  A stack is created before the loop with the first closed element, and all subsequent closed elements are pushed to the stack.  When an open element matches the element at the top of the stack, the stack is popped of that element.  When the stack length is zero, the loop ends.  The number of iterations is counted, and the outer loop skips that many iterations, i.e. the irrelevant subtree is culled.

This is what the code looks like:

 

This becomes more complex when we consider tags with multiple siblings of the same tag name.

nth-child Heuristic

When an ancestor of a path node has multiple tag names of the same type preceding it, the selector needs to specify the index of the path node.  The rationale for this is more easily understood by re-examining the charts from above:

In a table this looks like the following:

tag name index type include? path
p 0 open yes p
p 1 close no p:nth-child(2)
strong 2 close no p:nth-child(2)
strong 3 open no p:nth-child(2)
img 4 self-closing no p:nth-child(2)
p 5 open no p:nth-child(2)
body 6 open yes body > p:nth-child(2)

While the CSS Selector would not include the second <p> tag, the DOM tree would still grab both <p> tags unless a :nth-child selector is used.  The desired tree looks like this:

The code only needs minor modification.  Each token initializes an index value of 1.  The index increments when a subtree cull operation is started and the element that started the subtree cull is the same tag type as the nextElement to be pushed.

Conclusions

Stringification of an HTML document is sometimes necessary to reduce cost of tree traversal when the tree must be traversed from the root element and the search subjects of a traversal are sparse.  Generating CSS Selectors from a mouse event or given a stringified HTML document is handy for multiple popular use cases.

It’s important to consider the overall method of converting a tree into a string is useful for other popular data formats like JSON, as well.  Often, it’s cheaper to stringify a tree and search it than to traverse it and search it.  The implementation of a string-search is more easily encapsulated and straightforward than an iterative search as well.

Code

Here is the code covered in this post: