I found the culprit! I was setting both m24 and m43 to a value of one. But as you can see in this example, only one element of the matrix should have a non zero constant value.
Since m43 is the only element of the two that affects the Z axis, it has to be incorrect one. And indeed, after setting it to zero: