Make sure to associate your route table with the S3 Gateway Endpoint — it took me a while to realize that was the issue.