The problem was in fop.xconf not loading correctly.
I used to load it like this:

File configFile = new File("fop.xconf");
if (!configFile.exists()) {
    System.out.println("Config file not found: " + configFile.getAbsolutePath());
}

FopFactoryBuilder factoryBuilder = new FopFactoryBuilder(configFile.toURI());

In the end it would never load it.
So when you add this next code snippet behind the code above, it loads it correctly:

DefaultConfigurationBuilder cfgBuilder = new DefaultConfigurationBuilder();
Configuration cfg = cfgBuilder.buildFromFile(configFile);

factoryBuilder.setConfiguration(cfg);

FopFactory fopFactory = factoryBuilder.build();

This ended up fixing the whole thing.