After a lot of testing I think I now understand what is going on. I'm posting this as an answer as I think it is probably correct. If/when I get around to implementing a solution I'll come back and mark it as the correct answer.
When the user logs in to our IdentityServer it issues the user with a session cookie so that if they attempt to use any client, IdentityServer will know they are logged in and can issue the relevant tokens without asking them to log in again. The user could use Client A and Client B for a while (from my example in the OP), each of which then issue their own session cookies to the user.
If the user stops using Client B for a while, but continues to use Client A, then the session cookies (or the tokens within them) for Client B and the IdentityServer expire due to the idle timeout.
So if the user attempts to use Client B again, this client thinks they are unauthenticated and redirects to IdentityServer which also thinks they are unauthenticated and prompts them to log in. They are still able to carry on using Client A as its session cookie has been kept alive by its continual use.
This might be what @tore-nestenius was getting at in his comment on the OP, but he left a lot unsaid so I'm not sure if that was just a punt or if he'd recognised all this and jumped straight to a possible solution without an explanation.
I think the solution to this problem will involve storing session information for IdentityServer (and maybe also the client apps) on the server instead of in cookies. Which could enable use of client apps to cause the IdentityServer session to stay alive until 20 minutes after the last use of any client app. It could also provide many other advantages like being able to more easily track who is logged in to what and when using a centralised system.
Duende's Server Side Sessions Inactivity Timeout feature might offer a solution to my problem. The suggestion from @tore-nestenius on how to build a custom server side SessionStore or something based on it, might also do the job.