I solved this by not using a reference in the struct and using the Option::take function to move it out.

self.scanner = Some(Scanner::new(source));
self.chunk = Some(Chunk::new());
...
self.scanner = None;
let chunk = self.chunk.take().expect("How'd the chunk disappear, something really bad happend ig?");
Ok(chunk)

This was pretty self-explanatory and I should've held off on asking this question but it would still be good to see how someone else would do this?