Reimplement webpack raw-loader as babel-plugin-macros

• Published at January 18, 2018
• 7 mins reading timei

This weekend I was working on my side project to list and display all my future tech talks. This is an overengineered web app built using CRA and Spectacle, with a little bit of React Router (I'll write about it in another post). I want to be able to display talk information in both inside app and in GitHub (where I host the repository), so I decided to put talk title and description in README.md inside talk directory to prevent duplication.

preval.macro

My original solution to this problem was using preval.macro which uses babel-plugin-macro. Using preval is straightforward. Preval allows me to write JS code with filesystem access that will be executed on build step (via babel). It's really cool, you should try it yourself.

Basically you just import the macro, use it via tagged template, and put the code you want to execute as string.

import preval from 'preval.macro';
const markdown = preval`
  const fs = require('fs');
  const path = require('path');
  module.exports = fs.readFileSync(
    path.resolve(__dirname, './README.md'),
    { encoding: 'utf-8' }
  );
`;

This works but I don't like the boilerplate of requiring fs and path. It's also not reusable because I can't just pass file path.

raw-loader

Then I remember about webpack raw-loader. CRA uses webpack under the hood, so raw-loader should work. Unfortunately CRA doesn't yet have a solution regarding importing raw text file. I know this is unsupported method, but maybe still worth trying.

First I need to install raw-loader as devDependencies because CRA doesn't include it by default, then I need to use webpack loader require syntax because CRA doesn't allow me to extends webpack (for good reason). The result is slightly better than preval (in term of boilerplate).

const markdown = require('raw-loader!./README.md');

This fails the build of course, because CRA enable eslint rule to disallow webpack loader syntax. I need to disable it via eslint comment. Not pretty.

Then I had another use case for importing code example to spectacle code slide. This is where it gets weird. Somehow, importing it using raw-loader causes some newlines to be removed unintentionally. This is really frustrating because I need to annotate the exact line number to highlight some parts of code. This inconsistencies of removing newlines means I need to know the line number AFTER it was removed. Not pretty.

Then I was thinking of creating my own babel-plugin-macros to replace preval and raw-loader. I searched it first, but no luck.

It's time for another side project.

raw.macro

This is the first time I write macro using babel-plugin-macros, so I didn't have a clue what to do. I do know AST a little, but still. Fortunately I have a lot of inspirations from Kent C. Dodds. He has a lot of repositories using babel macro. (of course, he was the author of babel-plugin-macros).

I saw preval.macro codebase and learn basic macro creation, but the functionality is quite different, so not much to copy/paste (sorry Kent). Then I saw import-all.macro codebase and it's pretty similar to what I want to achieve. A function accepting arguments that gets compiled to another thing.

First step of creating macros are making sure all references in codebase get replaced. babel-plugin-macros provides createMacro function that accept function with some parameters:

import { createMacro } from 'babel-plugin-macros';
export default createMacro(rawMacro);
function createMacro({ references, state, babel }) {
  // do something
}

Our function will be given one argument (it's an object) that includes references, state, and babel. We use references to perform AST transform for all macro references in the codebase. state provides us with information regarding current processed file (we mostly interested with its filename, more on that later), and finally babel is mostly used to create new AST via babel-types available as babel.types (people usually rename it to t).

API requirement

We want to create macro like this

import raw from 'raw.macro';
const md = raw('./README.md');

That will be converted to this (depends on README.md content):

const md = '# Title\ndescription';

So we know that the only valid usage is function call (or CallExpression in the AST). First we need to process all references

function rawMacro({ references, state, babel }) {
  references.default.forEach((referencePath) => {
    if (referencePath.parentPath.type === 'CallExpression') {
      processReference({ referencePath, state, babel });
    } else {
      throw new Error('Invalid usage');
    }
  });
}

Then we need to get function arguments We only need one because the function only accepts one argument. We also need to make sure that the file path is deterministic (not a variable where we need to know the value).

function processReference({ referencePath, state, babel }) {
  let relativePath;
  try {
    relativePath = referencePath.parentPath
      .get('arguments')[0]
      .evaluate().value;
  } catch (err) {
    throw new Error('Value is not deterministic');
  }
}

Reading file contents

After getting function argument, we need to resolve its absolute path. This is where filename from state comes in handy. We can get directory of the filename using path, and merge the path with the relative path from previous step. We can use full path to get file contents via fs. We need to use sync method because macro needs to be resolved synchronously.

import path from 'path';
import fs from 'fs';
function processReference({ referencePath, state, babel }) {
  const dirname = path.dirname(state.file.opts.filename);
  const fullPath = path.resolve(dirname, relativePath);
  const content = fs.readFileSync(fullPath, { encoding: 'utf-8' });
}

Replacing AST

The last step is replace original AST. We'll use babel.types to create new AST. What we replace is the function call, with an expression containing string literal. We use replaceWith method on AST node that we want to replace.

function processReference({ referencePath, state, babel }) {
  const t = babel.types;
  referencePath.parentPath.replaceWith(
    t.expressionStatement(t.stringLiteral(content))
  );
});

That's it. We only need a few lines of code to replicate webpack raw-loader functionality.

Unit Testing

Finally we need to add some tests to make sure that our macro works as expected. The easiest way to unit test macro is by using babel-plugin-tester with snapshot enabled. We provides some code, and babel-plugin-tester will write a new snapshot containing our code and our transpiled code. You should learn about Snapshot Testing if you haven't already. It's really cool, useful, and will probably saves you a lot of time testing something.

const pluginTester = require('babel-plugin-tester');
const plugin = require('babel-plugin-macros');
const prettier = require('prettier');
pluginTester({
  plugin,
  snapshot: true,
  babelOptions: {
    filename: __filename,
  },
  formatResult(result) {
    return prettier.format(result, { trailingComma: 'es5' });
  },
  tests: {
    'correct usage': `
      import raw from '../macro';
      const md = raw('./fixtures/markdown.md');
    `,
  },
});

It will create this snapshot:

// Jest Snapshot v1, https://goo.gl/fbAQLP
exports[`macros correct usage: correct usage 1`] = `
"
import raw from '../macro';
const md = raw('./fixtures/markdown.md');
      ↓ ↓ ↓ ↓ ↓ ↓
const md = \\"# Title\\\\n\\\\nThis is the description\\\\n\\";
"
`;

As you see in the snapshot, the macro works as expected.

You can see end result in my repository: https://github.com/pveyes/raw.macro

I was really amazed on how much we can do with babel, especially with babel macros. The possibilities are endless. I think I'm going to write another babel macro soon.

​