I discovered the cause was that if the scanner C code identifies a token and returns it, TS will not backtrack before that token.