Try using ppo_trainer.dataloader.base_dataloader instead of ppo_trainer.dataloader
ppo_trainer.dataloader.base_dataloader
ppo_trainer.dataloader
Works for me with trl==0.11.3
trl==0.11.3