I found a post on Apple Developer CryptoKit to create JWT, which explains clearly how to decode the private key produced by openssl. Just as @dave_thompson_085's comments:

PKCS8 format for private keys, which contains both an algorithm identifier with optional parameters (here ed25519) PLUS the algoirthm-dependent key data

The point is we need to extract only the private key bytes after dumpasn1 the data!