I actually solved this and wanted to post the solution.
The fix was to convert to "SliverGridDelegateWithMaxCrossAxisExtent) and set the "maxCrossAxisExtent" to the width of the image, an then leave childAspectRatio as the width/height of the image.