I had to fix it directly on the prefixIcon widget, and Transform.scale() did the job for me.

CustomTextFormField(
  prefixIcon: Transform.scale(
    scale: 0.5,
    child: SvgPicture.asset(
      SvgPaths.calendar,
      height: 20.0,
      width: 20.0,
    ),
  ),
  isDense: false,
  hint: "Select Date",
)