I have fixed the error. I just used the github link. Here are my requirements from buildozer.spec:

requirements=python3,kivy==2.3.0,https://github.com/kivymd/KivyMD/archive/master.zip

It also turns out that buildozer decided to only use kivymd version 1.3.0 even though I deleted it once. Had to delete the .buildozer directory from my project directory. That caused problems down the line, so if anyone has the same issue, try to delete things properly. There should be a command in buildozer that cleans everything (although that will require another installation of all the components)