Turns out the issue seems to be in my Github workflow, not in Azure.

I had not included:

ARM_USE_OIDC: true

in the Github workflow.

Adding this has allowed the workflow to successfully run terraform init and create a state file in the Azure storage account.

Many thanks.