if you are using gemini-flash try switching to gemini-pro, Flash's tool calling is unstable and is probably the cause of your agent issue.