As mentioned here, we can just run apk add aws-cli to install AWS CLI v2 directly on top of Node image alpine version.
apk add aws-cli