Looks like the issue is some time-series had NaN in them. When Prometheus sum's a value with an NaN, it gives an NaN.

Answer is to add a >0 to the query, so:

sum by(topic) (kafka_consumer_fetch_manager_records_lag{application="abc123", topic="xyz"} >0 )

Found the answer in another post:

• prometheus sum one NaN value result into NaN. How to avoid it?