There are two views that contain firewall related data:
In the master database there is the sys.firewall_rules view
In both the master and the actual client databases there is a sys.database_firewall_rules view
The list of firewall rules you see in the Azure Portal corresponds with the sys.firewall_rules view in the master database.
To set the rules you can see in the Portal through t-sql you'd use, in the master database:
sys.sp_set_firewall_rule
I was trying to delete the greyed out rule with sys.sp_delete_database_firewall_rule. That's why it didn't work. I should have used sys.sp_delete_firewall_rule
EXECUTE sys.sp_delete_firewall_rule @name='name of rule to delete'
My mistake was that I was trying to remove a server firewall rule with the sp to delete a database firewall rule.
Compare: https://learn.microsoft.com/en-us/sql/relational-databases/system-stored-procedures/sp-delete-firewall-rule-azure-sql-database?view=azuresqldb-current
To: https://learn.microsoft.com/en-us/sql/relational-databases/system-stored-procedures/sp-delete-database-firewall-rule-azure-sql-database?view=azuresqldb-current
When I used sys.sp_delete_firewall_rule it all worked perfectly.