Using @supersize's answer, to keep it alongside my other keybinding overrides I added this to VSCode's user settings (JSON):

"vim.normalModeKeyBindingsNonRecursive": [
    {
        "before": ["<C-o>"],
        "commands": ["workbench.action.navigateBack"]
    },
    {
        "before": ["<C-i>"],
        "commands": ["workbench.action.navigateForward"]
    },
    // other overrides here
]

note: normalModeKeyBindingsNonRecursive is mentioned in VSCodeVim's README.md