Found a result that works for my use after looking into how rsync calls itself in an SSH session. In (Open)SSH you'd want the user to login like usual, with a shell, and you can override the command that'll be executed in that shell through the public key string (for OpenSSH, the AuthorizedKeysCommand executable is used to provide the string).

For the client pulling, the server is in --sender mode:

command="rsync --server --sender . 'test-file-1' 'test-file-2'" ssh-id25519 AAAA...

A client can then do:

$ rsync user@hostname:/ destination-dir/

If a client tries to 'push' files to the server, it results in an error. If a client does provide a different file list, the file list is overridden with the server-side file list.

I will be looking into possible security problems with overriding the override command, whether that's possible, otherwise people have direct access to the shell. For my case, the user is auto-generated and it cannot read anything outside its directory due to very restricted permissions. Root is also /sbin/nologin. If there's something that I'm missing in that regard, please tell.

If a user tries to plainly connect with ssh, it also starts up rsync --server --sender, waiting for an input. In that case, at least, the file list is already passed through so users cannot read other files.