This sounds like an example of the Branch Predictor making a mistake.

Without the `if` statement, the code has no branches. Add in some branches and the branch predictor has to start guessing which branch will run. And sometimes it gets it wrong, causing a performance penalty.