You can implement this using aion-torch, a library developed specifically to stabilize very deep PyTorch Transformers by replacing static residual connections with adaptive ones. It handles the scaling mathematics automatically to maintain gradient stability, often eliminating the need for aggressive gradient clipping or extremely low learning rates. Its new, but you can give a try and I will be happy to hear a feedback