In trying to narrow down the problem, I realized that there's probably a problem with Java and the (latest? version of) MacOS. Indeed, the following snippet seems to indicate that not all Locale's work. In this case, Locale.FRANCE doesn't work (the “Cancel”, “No”, “Yes” buttons remain in English), whereas Locale.GERMANY does. The initial problem I described may be related to this.

import java.util.Locale;
import javax.swing.JOptionPane;

public class Test {
    public static void main(String[] args) {
        
        //Locale.setDefault(Locale.FRANCE); // Doesn't work
        Locale.setDefault(Locale.GERMANY); // Works !
        
        JOptionPane.showConfirmDialog(null, "Message");
    }

}