Was in the same situation but managed to solve it (but on a linux vm as runner agent). i manage to sole it perfectly by doing this:
# login to az devops
az config set extension.use_dynamic_install=yes_without_prompt
echo $(System.AccessToken) | az devops login --organization "$(System.CollectionUri)"
# get the variable group id
group_id=$(az pipelines variable-group list --project "$(System.TeamProject)" --top ${{ parameters.search_top_n }} \
--query-order ${{ parameters.search_order }} --output table | grep ${{ parameters.variable_group_name }} | cut -d' ' -f1)
# create or update the variable
az pipelines variable-group variable create --project "$(System.TeamProject)" --group-id ${group_id} --name ${{ parameters.variable_key }} \
--value "${{ parameters.variable_value }}" --secret ${{ parameters.is_secret }} --output table || \
az pipelines variable-group variable update --project "$(System.TeamProject)" --group-id ${group_id} --name ${{ parameters.variable_key }} \
--value "${{ parameters.variable_value }}" --secret ${{ parameters.is_secret }} --output table
# logout from az devops
az devops logout
These are some links that make it easier to understand.
Having no permission for updating Variable Group via Azure DevOps REST API from running Pipeline
https://learn.microsoft.com/en-us/azure/devops/pipelines/build/variables?view=azure-devops&tabs=yaml