You are asking a really tough question that took me a while to understand. Your best bet is reading about change of measures, understanding the Girsanov theorem, the Radon-Nicodym derivative and what happens when you change measure; Ito's Lemma would also be useful.

I will attempt to explain, but I may fail miserably. The market tells you that it guesses how much inflation will be on average between now and year 3. It also guesses how much inflation will be on average between now and year 4. How inflation will actually pan out, the market tells you it doesn't know. Different models will give you different inflation convexities, depending on how you assume that inflation develops over time. But the common point is that all models will tell you that, when you look at the horizon at the end of year 4, the inflation at year 3 (as well as any other point, apart from now) is uncertain. To get the mean inflation at year 3, you then need to apply an adjustment to allow for the volatility over that time (as well as the volatility on forward interest rates I think).

The reason why you don't get that adjustment when you look straight at year 3 is because you use a different model/set of assumptions that target exactly year 3; not before not after. If you use these assumptions, you then would not know what happens in year 4.