It goes backwards (ctrl+o) or forwards (ctrl+i) in history of your jump commands. A jump command is any command that moves the cursor multiple lines away or in another file. :h jumplist will give you a much more detailed explanation but most important would be :jumps or :ju.