If anyone stumbles upon this question, I got it to work by setting architecture: lambda.Architecture.ARM_64 on my agent-invoked lambda.
architecture: lambda.Architecture.ARM_64