The best architecture to do this looks like a CDN or image proxy in front of S3 that allows for the signing and possibly even the encryption of URLs sent to the client. You are right to want to avoid sending the whole file by downloading it and passing it on to the client.

See imgproxy, an open source image proxy that enables resizing of images, signing and encrypting image URLs, serving images from an S3 bucket and more. There are alternatives that do similar things too.

If it were me, I'd use a server side implementation that encrypts/signs the URL and passes it to the client. I'd configure the headers on the response from the image proxy to determine how long the browser should cache the image.

I found this helpful blog post describing a basic set up of this kind of flow.