I was able to finally fix it by just deleting all pubspec.lock files in my project and then running flutter pub get again.

There seems to have been an issue with one flutter beta version, that put the dependency there and I could not get rid of it without cleaning up the pubspec files.