I had the exact problem as you and the exact same problem as the person linked to in Peter's comment. The answers in that post are great but I wanted more control, so I integrated a library called ScreenRecorderLib. I describe how I use it in this response. I also have a sample GitHub project that provides more detail.

Basically, add the ScreenRecorderLib package, then create a Recorder object during the test execution. After the test opens a browser, that browser window can be added to the Recorder object as a recordable source.

To answer your question and your particular problem, because the recorder is integrated into the project, you can start, pause, and stop the recording as needed, so you can have different behaviors for a failed test than a successful one. You'd probably have to come up with your own IO actions to move, delete, or rename the recording, but you would have control over starting and stopping the recorder.

And yes, one limitation is that the test cannot be executed in headless mode. there must be a window.