IDK if Rig might be any help here, but it simplifies Rust apps with LLM integrations, so could be worth checking out rig rs etc. if you're deep into Rust dev projects.
Ever worked with LLMs in Rust before?