When it comes down to numbers, then everybody is just "silent". Its a secret in data mining and huge conspiracy against end user ! Well i test a lot and i can only say, someone is lying big time ! All those tools aren't working the way are supposed to, results obtained in reality are clear evidence off that ! And all i hear in excuse of those tools used, "your data quality ins't good enough". What that sentence means in real, model cant learn from data (but it should due weights adjusted in model) !? Tools way to poor developed. Optuna in the wastes of HP space should find proper HP and model obtained should converge to that target. If u use the same file setup as in optuna with same classifier and target column u should get prety much the same results when test on rest file is used for that particular target column. Why those results differ so much its clue that i still figure it up. Its a math, that reproduces the same results then !? If not there is something wrong with math, weights aren't the same thus results are always worse. How to solve this i don't know. At the end i always manually tune models, find proper HP sometimes, that optuna newer finds, but it should. AI should do model HP search with all the needed metrics and evaluation plots and explanations. We don't have time to fool around with manualism's altogether. And i thought optuna will solve my problems too. Optuna at my experience newer delivers good models and i try out all the optionable metrics in optuna even custom balanced AUC original conditions formula as customizey by my ovn. Optuna simply isn't up the job. Issues for that can be very vast and all u hear is all philosophizing about it.