I stumbled across a couple of posts FAILED_PRECONDITION - Gmail API which provided an answer that worked for my case:

This line from the google code sample:

credentials = ServiceAccountCredentials.fromStream(stream).createScoped(GmailScopes.GMAIL_SEND);

needs to change to this:

ServiceAccountCredentials.fromStream(stream).createScoped(GmailScopes.GMAIL_SEND).createDelegated(workspaceEmailAddress);

The .createDelegate(workspaceEmailAddress) apparently ensures that the impersonation is happening at the right place.

Also for a bonus, to send an HTML formatted email, we need to change this line from the google sample code:

email.setText(htmlBodyText);

to this:

email.setContent(htmlBodyText, "text/html");