When it comes to changing the player's actual nametag, you have already discovered some of the only methods of doing this: by altering the protocol and using reflection. However, in most use-cases these methods are unsuitable as they change the behaviour of other functionalities, like how you encountered issues with the name not appearing in tab-completions, plugin commands, etc. If you change a players name on the server, it will essentially change everywhere.

To achieve the result of a custom nametag, as you have seen on servers such as Hypixel, you can utilise a nifty workaround. Instead of changing the player name, you can programmatically create an invisible armour stand at the player's position and alter the nametag of the armour stand entity instead, creating an illusion of a custom player nametag, when in reality you're looking at an invisible armour stand's nametag instead.

It can be quite difficult to implement this yourself, however, and I would recommend you use a library which will handle all the armour stand trickery for you. Try utilising Neznamy's TAB resource (Developer API).