I tried to use plotly and that was able to give me the result I was looking for -

import plotly.express as px
autompg_multi_index = autompg.query("yr<=80").groupby(['yr', 'origin'])['mpg'].mean().reset_index()
fig = px.bar(autompg_multi_index, x='yr', y='mpg', color='origin', barmode='group', text='mpg')
fig.update_traces(texttemplate='%{text:.2f}', textposition='outside')
fig.show()

Grouped bar chart with labels on top with 2 places after decimal point.

My solution is to abandon hvplot and migrate to plotly.