- What are the flaws of the above analysis?
The main flaw of such analysis is that you will account any delay in the execution to communication although the time is often not used to communicate but rather to wait for other processes. Assume rank 1 in a block-synchronous execution (barriers/all-to-all between the phases) takes 10% longer for each computation phase than the other two processes. Your analysis will indicate 7% of time is spent in barrier/all-to-all communication. To distinguish different causes of waiting time in parallel execution, Rosas et al. separate such influences as fundamental performance factors (https://doi.org/10.14529/jsfi140201).
- Assuming the bidirectional ring messaging has some inherent synchronization overhead, is this value also captured by the above equation e.g., by the area of the waiting functions?
Any inherent waiting time, whether it's overhead or waiting for other processes to get ready will end up in the timing of the wait function. If you are interested in actual communication overhead, you should compare measured execution time with theoretical execution time on an ideal network (0 latency, infinite bandwidth). The difference is the cost of executing the program on the actual network. Above work calls this cost the transfer efficiency factor. https://github.com/RWTH-HPC/OTF-CPT is a tool to measure transfer efficiency and the other performance factors during execution of an MPI(+OpenMP) program.
- Is there any overhead related to communication that the above analysis overlooks?
To allow the MPI implementation to make progress in the background, some implementations start an extra background thread. While such thread shouldn't spend too many cpu cycles, the time will most probably be accounted as compute time, if you fill the compute node with compute processes. If you leave some cores empty for OS and such background threads, this shouldn't be a concern.