I found the cause. It looks a trivial issue but not easy to realize.
First, I will explain each command why it works or doesn't work.
The following works because only SSH shell access is related, nothing to do with git.
ssh git@myserver
The following works because git does not use SSH.
git ls-remote http://ip:3000/user1/repo1.git
The following works because git+SSH loads "~/.ssh/id_ed25519" implicitly that I mistakenly thought the key should be "id_ed25519_repo1", in addition the key "id_ed25519" was configured via gitea web UI previously.
git ls-remote git@ip:user1/repo1.git
The following do not work because git+SSH loads "~/.ssh/id_ed25519_repo1" explicitly - the key was added to authorized_keys manually.
git ls-remote myserver:user1/repo1.git git ls-remote git@myserver:user1/repo1.git
So just adding a bare entry to the file like below, accessing "ssh user1@myserver" or "ssh user1@real-ip" will work well, but git+SSH absolutely does not work.
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIvf4l5RjqWL+kOnxpqhhGAIcIkWVSHqLbgkAzMAlYGm user1@domain
The reason is the missing a part that links SSH key to the git operations that explains why SSH auth is OK but git does not recognized the repo path. So the correct syntax to connect git to SSH should look like below:
command="/usr/local/bin/gitea --config=/etc/gitea/app.ini serv key-6",no-port-forwarding,no-X11-forwarding,no-user-rc,no-pty,restrict ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIvf4l5RjqWL+kOnxpqhhGAIcIkWVSHqLbgkAzMAlYGm user1@domain
It is quite long to manually edit, it'd better to let gitea adding that for us via web UI. But one issue appears, since the "command" comes in, the SSH shell access using "user1" becomes impossible. I don't know how to enable access via both git+SSH and SSH for the same user. My solution is to create a new key for pure SSH access or consider enable the PasswordAuthentication option.
Notes I want to share: