Unboxing webpack — making sense of loaders and plugins
In 2016 I traveled through Australia for a few weeks and tried surfing for the first time in my life. I loved it, but I had an uneasy feeling whenever I went into the water. There were jellyfish under the surface, and after I had watched a few unlucky bathers, I knew that touching them hurt.
Webpack, to me, is a bit like these jellyfish. It is part of the development environment and has reasons to be there, but we usually try to stay away from each other because we know that any contact will be painful for both of us. Unfortunately, many of the technical problems I faced at work lately seemed to be related to webpack somehow, so avoiding contact was rarely an option. I usually reached out to a colleague for help, and we solved the problem with google and a lot of experimentation.
For me, most of the confusion around webpack stemmed from the fact that I had never set it up myself and didn’t really understand what was going on under the hood. I think that’s the case for many people because you usually don’t configure webpack from scratch. For example, when you start a new Vue project, the Vue CLI will preconfigure webpack for you. I think the same is true for React. That’s convenient because you don’t want to spend half an hour setting up webpack every time you start a new project. But if you have not set things up yourself at least once, this “magic” is difficult to understand once you have to customize it. So let’s look under the hood and set up webpack ourselves. We won’t stitch a bunch of existing plugins together, though. We will build them from scratch 💪.
The problem webpack tries to solve
Let’s start at the beginning and figure out what webpack actually is. The webpack documentation describes webpack as a “static module bundler for modern JavaScript applications.” That sounds great, but if you don’t know much about bundlers, then, above all, this sounds very abstract. In simple terms, webpack will look at your JavaScript code, figure out all of its dependencies and bundle them together into a single directory, as one or multiple files.
A few years ago, this wasn’t necessary because JavaScript applications usually weren’t that large anyway. You could just include a couple of <script> tags in your HTML. But nowadays, applications have become so large that your head would explode if you tried to include all of your JavaScript files manually and in the right order. But webpack goes even further and makes it possible to bundle images, stylesheets and even your HTML. It does all of this through loaders and plugins, making webpack extremely customizable – and sometimes also causing your head to explode.
The project setup
We will build a simple frontend app, including some HTML, CSS and JavaScript, and bundle all of it with webpack. This will be a tiny project, but it should suffice to illustrate some of the concepts behind webpack. Let’s set up the project folder and install webpack and the webpack-cli.
mkdir webpack-from-scratch
cd webpack-from-scratch
mkdir src
echo "console.log('Hello world');" > src/index.js
yarn add -D webpack webpack-cli
Now we can run yarn webpack bundle from the terminal, and webpack will bundle our index.js and output it to dist/main.js. How does webpack know which file to bundle for us? By convention. The default webpack config assumes that there exists a src/index.js file and uses it as the bundle’s entry point. The “entry point” is the source file from which webpack starts to build the dependency tree. You can change the entry point and even add multiple entry points, but we won’t have to for our example. Let’s inspect the content of our bundled app:
// dist/main.js
console.log('Hello world');
Of course, webpack doesn’t really add any value to our project yet. We could have just as easily copied the index.js file over ourselves. But we will get to that. First, let’s add an HTML file to load our JavaScript code. We will look at how to bundle it automatically later, but for now, let’s save the file in the dist/ folder.
<!-- dist/index.html -->
<html>
<head>
<script type="text/javascript" src="main.js"></script>
</head>
<body>
<div id="app">Hello world app</div>
</body>
</html>
Open the file in your browser, and you should see “Hello world” logged to your developer console. Next, let’s make our JavaScript actually do something. How about showing the current time on our website? To keep the code modular, create a new file src/dateFormat.js that exports a single function:
// src/dateFormat.js
export const weekdayWithTime = (date) => {
const WEEKDAYS = ['Sunday', 'Monday', 'Tuesday', 'Wednesday', 'Thursday', 'Friday', 'Saturday'];
const currentWeekday = WEEKDAYS[date.getDay()];
const time = `${date.getHours()}:${date.getMinutes()}:${date.getSeconds()}`;
return `${currentWeekday}, ${time}`;
};
We import the function in the index.js to display the current weekday and time on the web page and update it every second.
// src/index.js
import { weekdayWithTime } from './dateFormat';
setInterval(() => {
const app = document.querySelector('#app');
if (app) {
app.innerHTML = weekdayWithTime(new Date());
}
}, 1000);
Run yarn webpack bundle to rebuild the app, then open the index.html again. You should see a clock that’s updated every second. When you open the bundled file again, you will see that webpack has included the code from src/dateFormat.js. It has also minified the code.
Now we would like to add some styling to our page. In the old days, we would have created the stylesheet and added it to the HTML ourselves. But letting webpack handle all of our assets has a few benefits: Each module can explicitly import the assets it depends on. That makes it easier for developers to keep track of the dependencies, and it allows webpack to only include the stuff in the bundle that’s actually needed. For example, if your website has five pages that all use different JavaScript and stylesheets, you can create five bundles, each with its own dependency tree. This might not sound like a big deal as long as your codebase is small, but eventually, your app is going to grow. Excluding code that’s not needed might make the difference of a few milliseconds during page load, which can be an eternity on the web.
Loading stylesheets with webpack
We will save the styles in src/style.css. Let’s give the body a bright background color, center the text and change the font family and size:
/* src/style.css */
body {
background-color: #F32B4D;
display: flex;
justify-content: center;
align-items: center;
}
#app {
font-family: Helvetica, Arial, sans-serif;
color: #FFFFFF;
font-size: 72px;
font-weight: bold;
}
When we import the stylesheet in the index.js file, webpack will add it to the dependency tree and try to load it during compilation.
// src/index.js
import { weekdayWithTime } from './dateFormat';
import stylesheetPath from './style.css';
// ...
Running yarn webpack bundle will give us the following error:
You may need an appropriate loader to handle this file type, currently no loaders are configured to process this file. See https://webpack.js.org/concepts#loaders
Webpack doesn’t know what to do with the stylesheet. We will have to add an appropriate loader to fix this.
Creating our own stylesheet loader
When webpack encounters an import statement, it will load the imported file’s content into an UTF-8 string. A webpack loader is just a function. It takes the string as an argument, optionally applies transformations to it, and returns the result. It really is that simple. You can build a loader in just a few lines of JavaScript:
module.exports = (source) => {
return source.replace('foo', 'bar');
}
By default, webpack only knows how to import JavaScript files. But you can configure webpack to use different loaders for other file types, and you can chain multiple loaders for the same file type. Then, each loader will transform the resource and pass the resulting output to the next loader in the chain. Loaders can transform the input in any way they like, but the last loader in the chain must produce a valid JavaScript module, which will then be executed instead of the original resource.
We want our loader to copy the stylesheet to our build directory. Then it should export the path to the generated file instead of returning the stylesheet content. We can use the emitFile function from webpack’s loader context to emit the stylesheet to the build directory. Let’s create a stylesheet loader in webpack/CssLoader.js:
// webpack/CssLoader.js
const path = require('path');
const randomFilename = (ext) => `${new Date().getTime()}-${Math.random()}.${ext}`;
module.exports = function(source) {
// emit a file with a random file name and the stylesheet content
// to the output directory
const outputFilename = randomFilename('css');
this.emitFile(outputFilename, source);
// return a module that exports the path to the file
return `module.exports = '${outputFilename}';`;
};
This will output a .css file with a random file name and return a module that exports the file’s path. We use a random file name to avoid name clashes when we import multiple stylesheets with the same basename. It’s crucial that you define the loader as a normal function and not as an arrow function because the latter wouldn’t have access to the loader context through this.
Initially, I tried to return the plain file path from the loader, expecting to be able to import it. That didn’t work because a file path is not valid JavaScript code. Remember what I said earlier, that the last loader in the chain has to return a valid JavaScript module? Think of it like this: When you import a module, webpack will evaluate the module’s source code in place of the import and expose the module’s exports in the current scope.
- import foo from 'path/to/bar';
+ const foo = () => {
+ // module code evaluated here
+ return module.exports;
+ }();
Given the example above, returning the plain file path would produce this:
- import stylesheetPath from './dateFormat';
+ const stylesheetPath = () => { dist/1612259960631-0.19359674520973624.css }();
Which is not valid JavaScript code. First, there are no exports, so the evaluated code doesn’t return any value. Second, the file path is not enquoted, resulting in a syntax error. Of course, this is a very simplified (and probably somewhat wrong) explanation of the complicated stuff that goes on under the hood, but I hope it helps to understand what the output of a loader has to look like.
By the way, it’s easy to change the loader to work with other file types, images for example. Have a look at the webpack docs about raw loaders to see how to load resources without converting them to UTF-8 strings.
Now that we have created our loader, we need to tell webpack to use it. Let’s create a webpack configuration file:
touch webpack.config.js
And set up a loader rule for .css files:
// webpack.config.js
const path = require('path');
module.exports = {
module: {
rules: [{
test: /\.css$/, // use this rule for all files that end on .css
use: [{
loader: path.resolve(__dirname, 'webpack/CssLoader.js'),
}],
}],
},
};
Now we can run yarn webpack bundle without getting any errors. But we haven’t added the stylesheet to our HTML, so it’s not gonna do anything. Let’s use JavaScript to append the stylesheet to the DOM:
// src/index.js
import stylesheetPath from './style.css';
// ...
document.addEventListener('DOMContentLoaded', () => {
const tag = document.createElement('link');
tag.setAttribute('rel', 'stylesheet');
tag.setAttribute('type', 'text/css');
tag.setAttribute('href', stylesheetPath);
document.querySelector('head').appendChild(tag);
});
Bundle the app again, and you should see a beautiful, bright background 🎉.
Unfortunately, this has one drawback: The <link> tag gets added to the DOM by our JavaScript code during runtime, which means that it is loaded last, after the web page was rendered, and after the JavaScript code was executed. Ideally, we would like to append the <link> tag to our HTML during compilation. How do we do that? With our own plugin!
Building a plugin to generate HTML at compile time
Initially, we have created the index.html file and manually copied it to the dist/ folder. Of course, we could also go ahead and add a <link> tag for the stylesheet to that HTML file. But what if we have more than one stylesheet? This quickly becomes unmanageable. Wouldn’t it be great if webpack did this for us automatically? Can we do that with a loader as well?
Loaders don’t have information about other resources that the module depends on. We could create an HTML loader, but it wouldn’t be able to know which stylesheets it has to add to the template. Instead, we have to create a webpack plugin. Let’s examine how plugins work.
When we run the webpack bundle command, webpack runs a series of steps to generate the final bundle. It resolves dependencies, optimizes code, emits assets – the list is pretty long. A plugin allows us to hook into each of these steps to do additional work.
Unfortunately, even though writing a plugin itself isn’t very complicated, I had difficulty figuring out how all of the available hooks work and which one I had to tap into. The webpack docs include a list of available hooks, but the information on each individual hook is very limited. What helped me most was to dig into the source code of some of the webpack plugins.
The good news is that you rarely have to write a plugin yourself. There are many existing webpack plugins, and most of the stuff you usually want to do has already been figured out by someone else. For example, a plugin similar to what we are going to build (in fact, it’s much more powerful) is the html-webpack-plugin. But we’re here to take the cardbox apart, so sharpen your cutter.
A plugin is a JavaScript class with an apply method. The method receives a compiler instance as an argument, which exposes a list of hooks we can tap into. We need to hook into the compilation after webpack has emitted all assets because that allows us to get the list of stylesheets we need to append to the HTML template.
We tap into the thisCompilation hook, which in turn gives us access to the compilationContext. This, again, exposes several hooks we can tap into, one of which is the processAssets hook. The processAssets hook has multiple stages, and we hook into the stage that outputs additional assets. Did I mention that this stuff is pretty complex? Hopefully, the code will make this more clear. Let’s create the plugin in webpack/HtmlPlugin.js:
// webpack/HtmlPlugin.js
const fs = require('fs');
const path = require('path');
const webpack = require('webpack');
const createLinkTag = (filepath) => `<link rel="stylesheet" href="${filepath}">`;
const substituteTemplateVariables = (template, stylesheets) => {
const stylesheetTags = stylesheets.map(createLinkTag);
// the template uses a placeholder that we replace with the <link> tags
return template.replace('%stylesheets%', stylesheetTags.join('\n'));
};
class HtmlPlugin {
apply(compiler) {
compiler.hooks.thisCompilation.tap('HtmlPlugin', (compilation) => {
compilation.hooks.processAssets.tapAsync({
name: 'HtmlPlugin',
stage: webpack.Compilation.PROCESS_ASSETS_STAGE_ADDITIONAL,
}, (assets, callback) => {
// assets is an object where each key corresponds to the
// file path of an asset that webpack generated. We filter
// them to select only stylesheets.
const stylesheets = Object.keys(assets)
.filter((asset) => asset.endsWith('.css'));
// we load a template file that we populate with the <link> tags
const pathToTemplate = path.join(__dirname, 'HtmlPlugin.html');
const templateContent = fs.readFileSync(pathToTemplate, 'utf8');
const source = substituteTemplateVariables(templateContent, stylesheets);
// and emit the index.html asset
compilation.emitAsset('index.html', new webpack.sources.RawSource(source));
callback();
})
});
}
}
module.exports = HtmlPlugin;
The fact that there are nested hooks didn’t make this any less confusing. How did I know which hooks to tap into? I just looked at the source code of the html-webpack-plugin to see what they did. The webpack documentation on the processAssets hook is actually pretty good, but I don’t think I would have ever found the hook myself. I also ran many test compilations, where I just console.logged the hooks’ arguments to see what their interfaces are.
The hook uses an HTML template, which we haven’t created, yet. Let’s do that next:
<!-- webpack/HtmlPlugin.html -->
<html>
<head>
<!-- placeholder to be replaced with link tags -->
%stylesheets%
<script type="text/javascript" src="main.js"></script>
</head>
<body>
<div id="app"></div>
</body>
</html>
And finally, let’s add the plugin to our webpack config:
// webpack.config.js
const HtmlPlugin = require('./webpack/HtmlPlugin');
module.exports = {
// ...
plugins: [new HtmlPlugin()],
};
Unlike when we added our custom loader, we need to import the plugin, instantiate it and pass the plugin instance to webpack. Now you can remove the event listener from the index.js, but keep the stylesheet import. We no longer need the stylesheetPath, but without the import, webpack will not add the stylesheet to the dependency tree.
// src/index.js
- import stylesheetPath from './style.css';
+ import './style.css';
- document.addEventListener('DOMContentLoaded', () => {
- const tag = document.createElement('link');
- tag.setAttribute('rel', 'stylesheet');
- tag.setAttribute('type', 'text/css');
- tag.setAttribute('href', stylesheetPath);
- document.querySelector('head').appendChild(tag);
- });
When you run yarn webpack bundle again, it should emit three files to the output directory:
dist/
|- 1612259960631-0.19359674520973624.css
|- index.html
|- main.js
The index.html should still have bold text and the marvelous background color, but now the styles are loaded before the JavaScript.
Conclusion
Looking back, we have started with a single JavaScript file that we bundled with webpack. Then we’ve added a stylesheet and built our own webpack loader to import it. Lastly, we have developed a plugin that generates an HTML file and populates it with <link> tags to all of the bundle’s stylesheets.
Of course, you wouldn’t normally build this from scratch. There are more polished webpack loaders and plugins that can do much better, what we have done in only a few lines of code. But I hope that you have enjoyed the exercise to build things for yourself this time – I have at least. I’m also less afraid to plunge into the depths of webpack now, even though we still only scratched the surface. You can write a whole book about the things that webpack can do. Talking about books, I think the webpack book on survivejs.com is a great resource to learn more about webpack. Use it as a reference if you keep experimenting. For example, you could extend the plugin to have it automatically add the <script> tag that links to the JavaScript bundle. That’s useful if you decide to change the bundle name or if you configure webpack to output more than one bundle.