There is now an implementation of RoPE at torchtune.modules.RotaryPositionalEmbeddings; torchtune is authored by the official pytorch team.