I have set my --max-memory-restart to 1300MB (I have 2GB ram on VPS), and it seems to be more stable, now.
Those spikes aren't happening for 2 days straight. But if it happens again, I will try to upgrade my VPS specs to like 2 core and 4GB one.