Transpilers: How They Work and How To Build Your Own JS Transpiler
Now, In the JS world, the syntaxes have shifted a lot (still shifting), and much has been added. We have the
All these new releases of ES* comes with new features. Old browsers will become incompatible with websites built with a newer feature or newer ES.
If for example, Firefox v1 supports the A1 feature in ES5 when ES6 is released with the A2 feature, Firefox v1 won’t be able to support the A2 feature but a new Firefox V2 release will be built to support the A2 release so what happens to poor users still using the old Firefox v1?
This is where transpilers come to help.
What are Transpilers?
Compilers translate code from one language to another ex. Java to bytecode. Transpilers translate code to the same language
Transpilers transform the code of a language into another form of the same language. Like Java transpiler translates a form of Java code to another form of Java code.
A new addition to ES6 is the use of classes:
An old browser that doesn't support class will throw an incompatibility error. Another form of writing a class is the function keyword and prototype keyword. The above could be done using function:
So when the transpiler detects the browser doesn’t support the class, it would convert the Book class to its function version. It would parse through the code and output the function version.
So a transpiler scans through the source-code, generates tokens and ASTs from the source, and finally, an interpreter outputs the source.
The difference between an Interpreter and Compiler
An interpreter scans through the code, generates syntax trees and executes the instructions one after another.
For example, we have an array of statements:
A compiler transforms a language into another language, for example the C/C++ code to Machine code, JS code to bytecode.
The first step starts with generating tokens from the source code. Then, the tokens are transformed into a syntax tree called AST. The AST is parsed by the parser and fed to the output interpreter which prints out the corresponding form.
In this post, I’ll skip the generating tokens stage and the generating AST stage. We will start with the output interpreter
Using our former example:
Ok, we will build a transpiler that will transform the class to function.
First, we can represent classes in AST like this:
It will have properties clsName, that will hold the name of the class, methods an array that will hold the methods in the class.
Each method in the class will be represented like this:
It will have a name property that holds the name of the property, type, denotes the type of the method whether static or not, then body, that holds in array the statements of the method.
So our Book class would be represented like this:
Now we will write an output interpreter. Our interpreting will be based on the Visitor pattern.
Visitor pattern enables us to create a single class and write the implementation of the ASTs in the class and feed the class to the ASTs. Each AST will have a common method that the interpreter will call with the implementing class instance. Then the AST class would each call the corresponding method in the passed in implementing class that evaluates its AST.
We add a visit method to our ClassDecl class, the visit method will take a Visitor instance as param and call the corresponding method that implements its algorithm.
Let’s create the Visitor class first and add a method to evaluate ClassDecl AST:
Now we add the visit method to ClassDecl class:
Now we flesh out our methods in the Visitor class:
Now let’s run the interpreter:
Yay!! We have transpiled
Building Scanner and Parser
We have our AST class for transpiling classes to functions. In this section, we will write the scanner and parser.
The scanner will turn the text of JS code into tokens, then feed it to the parser, who will turn the tokens into a syntax tree. This syntax tree is what we did above which will be fed into the output interpreter to produce the JS code.
The Scanner class scans through the string passed in str and produces tokens. The tokenize method aggregates the string in s and passes it through a series of checks: Number check, RParen, and LParen check, alphanumeric check, keyword check, identifier check, and operator check. If any check passes it produces a token for the string.
If we pass in:
Running the Scanner:
We have the tokens, the tokens will now be passed to the parser which will produce the AST.
The parsing starts from the parse method. It takes in the tokens produced by the Scanner class, it parses through it, and generates the corresponding AST. We only added parsing for class declarations just for this article.
So when passed in the tokens we generated earlier
Then, using our Visitor class we implemented in the last section we will pass it the AST generated here.
We added a new method visitStatements which takes an array of statements, loops through them, and calls the statement visit method passing itself to it.
The AST generated by the Parser is returned in an array so that’s why we called the visitStatements and passed the array to it.
Let’s print the result:
Now let’s print everything to see the transition:
See the transitions. From the source to tokens to AST to the final output. We have successfully transpiled a JS code to another form of JS code.
Bringing all files together
Let’s make our implementation run like Babel. Instead of passing it a string, it should be able to be passed a file we want to transpile, like this:
to produce output test/cls-transpiled.js.
Create a project, cd into it, and initialize a Node environment:
See full code on GitHub:
Now in our index.js file, we need to get the inputs passed when running our project, we will use the process object. First, we will get the inputs from process.argv, then use the filesystem module fs to read the file into a buffer the pass in the buffer to the Scanner to start the transpiling. Then when done, the result is written to a file test/cls-transpiled.js.
Before we run this, let’s create a folder test and inside it create a file cls.js
Now we open the file and add the following contents to it:
Now we run our project:
We will see:
Logged in our console.
Also, cls-transpiled.js will be created in our test folder. If we open it we will see:
See the full-code of our transpiler on Github: https://github.com/philipszdavido/transpiler
We saw what transpilers are and what they do. We also practically demonstrated how they work by writing a transpiler!!
So whenever you use Babel, TypeScript, or Traceur, you know what happens underneath.
Like always, I’m an advocate of practical work. To deeply understand how transpilers work. I urge you to add to this project in this article. We made a transpiler to support classes in JS, you can add features of your own to JS and write your transpiler for it. For example, we have this awesome proposal in JS to add private fields/methods to JS classes, you can support that on your own by writing your transpiler for it without waiting for the official implementation :) It will be awesome, imagine being able to add anything you want to your favorite language, it's god-like.
Features you may want to add to this:
- Spread and Rest
- Arrow functions
- Getters and Setters
If you have any questions regarding this or anything I should add, correct or remove, feel free to comment, email, or DM me.
- Angular Compiler: I got inspiration from Angular compiler’s JS emitter on how to write mine for this article.
- Crafting Interpreters: Taught me how to write tokenizers and parsers. After going through series of compilers/interpreters tut on the web I got the idea and concept when I landed on craftinginterpreters.com, other resources skipped so many things and showed a very complex code without touching on how it was formed. Thank you.
If you are busy or lazy it's ok, try our weekly recap and we'll save your time