I ran into this issue when working with our code signing private key being stored in Azure KeyVault. The original pair of Private-Public keys was generated and stored using a Hardware Security Module (HSM), which was on-premise and not accessible to our Azure DevOps cloud-based build agents. A solution was to move the private key into Azure KeyVault using their Bring You Own Key (BYOK) mechanism, use signtool to create the file digest using the trusted code signing certificate, sign the digest using Azure KeyVault functions and then ingest the signature.

I then observed this issue.

The issue for me was that our code signing key pair was generated from an Elliptic Curve (P-384). The signed digest can not simply be returned to signtool "as-is".

The EC signed digest needs to be encoded using ASN.1 DER format. SEQUENCE(r,s) (RFC3279 DER Sequence). r and s are the first and second halves of the returned signed digest.

In the end I wrote a C++ application that performed the same functionality as signtool, using the SignSignerEx3 function in the Windows SDK. This allows the correct hashing of PE files to be done by Windows and a callback function called when the digest needs to be signed.

For those who might be following a similar journey, I found this out by reading the .NET code. There is a subtle difference between 'SignData' and 'SignHash' functions, with the later encoding the signed digest for EC keys or adding PCKS1 padding for RSA keys.

• EC - https://github.com/dotnet/runtime/blob/1d1bf92fcf43aa6981804dc53c5174445069c9e4/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/ECDsa.cs#L1095

• RSA - https://github.com/dotnet/runtime/blob/1d1bf92fcf43aa6981804dc53c5174445069c9e4/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/RSA.cs#L341