I finally got it working by following this comment https://stackoverflow.com/a/43684021/24823862
below is the code for anyone who wants to use it
from M2Crypto import RSA
import binascii
import hashlib
import base64
# Data to hash
data = 'helloworld'
# Compute SHA-1 hash
sha1_hash = hashlib.sha1(data.encode('utf-8')).digest()
# Convert SHA-1 hash to hex string
sha1_hash_str = binascii.hexlify(sha1_hash).decode('utf-8')
# Read private key
private_key = RSA.load_key('key.pem')
# Encrypt the SHA-1 hash string using private key
ciphertext = private_key.private_encrypt(sha1_hash_str.encode('utf-8'), RSA.pkcs1_padding)
#encrypted_message = str(base64.b64encode(ciphertext), 'utf8')
encrypted_message = binascii.hexlify(ciphertext).decode('utf-8')
print(encrypted_message)
generated hash
01540a01e7c372f4d1395221ec90f68a0f4dbc123af9d032b768fd141c0b0a6420e0dcf903739dd2729cfdbf81bcd9512cc39ad4bd26239eab23069fdaf4e6fe
I got M2 crypto running in an ubuntu v22.04.3 LTS machine with python v3.10.12 and followed these instruction
M2 crypto repo - https://gitlab.com/m2crypto/m2crypto/-/tree/master
instructions to install - https://gitlab.com/m2crypto/m2crypto/-/blob/master/INSTALL.rst
Attaching two useful stackoverflow posts that helped me