This could be the result of many issues, so try to change and test out different variations of the following:

• Learning Rate (Maybe an LR Scheduler)
• Optimizer
• Dataset Size

additionally, you can look into batch normalization.

Hope this helps