- How do I determine whether my program would benefit from setting a lower precision? Is this purely empirical?
By setting torch.set_float32_matmul_precision() you change the behavior (which includes performance and accuracy) of your matrix multiplications in FP32. If you know (via, e.g., profiling) your code is spending a reasonable amount of time multiplying matrices in FP32 and you have hardware that is able to perform mixed-precision matrix multiplications, you will probably see performance benefits in using the high or medium options. So, not purely empirical. In terms of accuracy, medium is the less accurate option. If you care about accuracy, you should have a way to measure it in your application even if you opt for using the higher option. It may be the case that FP32 (i.e., higher) is not accurate enough for your application.
- How do you determine which of the lower precisions ("high" or "medium") to use? Is this purely empirical? What's the suggested approach?
The option high, as the documentation says, enables the use of the type TF32 internally. As far as I know, this type is only supported by Nvidia GPUs, so the first limitation for you may be related to hardware. Although it seems the GPU you mentioned has it available. Option medium uses BF16. See https://blogs.nvidia.com/blog/tensorfloat-32-precision-format/ for a comparison between BF16 and TF32.
I suggest you measure the accuracy of your application regardless of the option you choose for torch.set_float32_matmul_precision(). If accuracy is under control, go with the fastest option, which I believe will be medium.