I decided to come up with a little Trait that allows for indexing with a TryInto<usize> instead of just usize. This simplifies the logic a lot:

fn main() {
    let vec = vec![1, 2, 3];
    let mask = [-1, 1];
    
    for i in 0..vec.len() {
        let left = vec.get_index(i as i32 + mask[0]).unwrap_or(&0);
        let right = vec.get_index(i as i32 + mask[1]).unwrap_or(&0);
        
        println!("Sum: {}", left - right);
    }
}

pub trait Indexable<I> {
    type Item;
    fn get_index(&self, index: I) -> Option<&Self::Item>;
}

impl<T, I> Indexable<I> for Vec<T>
where
    I: TryInto<usize>,
{
    type Item = T;
    fn get_index(&self, index: I) -> Option<&Self::Item> {
        let Ok(num) = index.try_into() else {
            return None;
        };
        self.get(num)
    }
}

playground

This is similar to a wrapper but it allows for more flexiblity. You could technically even create a wrapper type around Option<&mut T> so you can change an index without bound-checks even:

vec.get_index(index as i32).and_set(3);
assert_eq!(Some(3), vec.get(index));

Simliar to the entry()-Method in HashMap. Anyways, thank you all for the great ideas!