I was using the EsBuild instructions when I should have been using the Webpack instructions.

The webpack instructions work for the most part but the documentation is missing a variable declaration

module.exports = (config, options, context) => {
  config.plugins = config.plugins || []; // Add this line in. Without it, it'll be undefined
  config.mode = process.env.NODE_ENV || config.mode;
  config.plugins.push(new webpack.DefinePlugin(getClientEnvironment()));
  return config;
};

After following the Webpack instructions and adding the commented line above, I was able to get env vars working.