You can add a prompt that enforces the model to prioritize earlier answers to ensure consistency. For example, you may ask the model to validate if its new answer conflicts with its prior knowledge, and only change the answer if its new input is significantly more reliable.

A possible prompt template like “Are you confident if this new answer is correct based on your knowledge?”.

However, when generating responses, you can adjust the model’s temperature and sampling strategies. A higher temperature often leads to more varied outputs, while a lower temperature results in more deterministic answers. By controlling these parameters, you can increase the model's confidence.