I think this issue might also be caused by the chat template used with the Llama model. For example, the LLaMA 3.2 template includes the instruction:

"Given the following functions, please respond with a JSON for a function call with its proper arguments that best answers the given prompt."

Leaving it unchanged can effectively force the model to call tools, even when unnecessary.

Note that the template of the llama3-groq-tool-use model @LaurieYoung mentioned is less "agressive":

You may call one or more functions to assist with the user query.