There has been a bug that evaluation metrics created by make_genai_metric_from_prompt cannot be used for mlflow.evaluate. A PR (https://github.com/mlflow/mlflow/pull/16960) has been filed to fix the issue, which should be included in the next minor release. Sorry for the inconvenience, and thank you for the report. From now one, we would appreciate it if you could file an issue at https://github.com/mlflow/mlflow/issues when you find an issue. Thank you!