After throughly reviewing all involved settings, I was not being able to list labels using impersonation.

Turns out, Google has enabled an Api Explorer in the same Api reference webpage, so you can try the endpoint after a simple OAuth authentication.

After some trial and error, in my case the problem was I had to set

labels.list.useAdminAccess = true