Is there any method to connect two cellular router directly in IP layer? more straightful, connect server and client directly in IP layer bypass the public IP middle server. the network topology i want show below.

Yes, if they have public IP addresses. At least the "server" device, that is.

This is not anything specific to cellular (well, not anymore). They both are on the same network in a sense – the Internet – but the respective ISPs might not be giving them a dedicated IPv4 address, instead choosing to put them behind CGNAT (and if they do give a dedicated address, they may still be blocking inbound connections for a variety of reasons). It's just far more common on cellular, but these days has become just as widespread even for residential connections.

So you need to contact your operator (for the "server" device) and find out what the options are. Sometimes they provide an IPv4 address as an addon, sometimes they have specific plans that include one. Sometimes if the real operator doesn't do this, you may be able to find IoT-oriented MVNOs which do. Once your "server" SIM card has been assigned a dedicated IPv4 address, just connect to that.

Increasingly often, ISPs do provide IPv6 addresses by default (as there is more of those available to give out), so if your router and client/server software support IPv6 (ought to, in 2025!) then you might be able to make a direct connection using those addresses instead of IPv4 – though a cellular ISP may still decide to block incoming connections and you may still have to contact them regardless.