Since I can't simply comment, still wanted to share an additional couple endpoints that helped out greatly. @bright's answer is still right on the money, but in addition to the tags, I wanted to know the underlying agent's capabilities.
GET https://dev.azure.com/{organization}/_apis/distributedtask/pools/{poolId}/agents?api-version=7.1&includeCapabilities=true
Pool id can be found from "id" property returned from:
GET https://dev.azure.com/{organization}/{project}/_apis/pipelines/environments/{environmentId}/providers/virtualmachines/pool?api-version=7.2-preview.1
P.S. That /agents endpoint has a demands query property where you can pre-filter your list of agents with certain capabilities (i.e. demands=Agent.OS), although it doesn't seem to check against the value itself, just if the capability exists by name. It will also return a collection of capabilities with just the demands you specified, which will lighten the returned payload quite a bit in most cases, which is exactly what I wanted.