Latency or packet loss can cause desynchronization, where multiple peers claim one thing and the rest claim another thing. None of them can agree on what happened since there is no ground truth or any other entity that can validate. It's like asking to hold a game of Football without a referee. Yes players can moderate each other, but all it needs is enough people to blink at the same time, to disagree on what events occurred in the match. You may start up in a synchronized state, but eventually, peers will diverge. It's just a byproduct of entropy.

Besides this obvious issue, P2P networks are not very secure, and it would be very troublesome to keep hackers banned, they can just reconnect without a server moderating. Sure you could keep a copy of banned accounts, but again, you have no way to validate bans. Also it's very easy for someone to flood the network with fake accounts, to the point where a distributing a copy of the list of bans becomes infeasible.

Finally, a hacker can send packets with fake position.

I don't want any latency of adding in a server

Everything i mentioned above is conflict issues, which you could still resolve. However, traversing the network and validating information from every peer will add extreme latency. Even if you parallelize the network, every peer's latency will depend on the slowest of the group.

In conclusion: Both systems will introduce latency, but a Server can provide conflict resolution and better moderation, but a P2P network will have a lot of difficulty replicating the same behaviour, and implementing and designing that system will be strenuous.