You can remove the minimum height, by using Modifier.heightIn(min = 0.dp). This should fix your "default padding" issue.
Modifier.heightIn(min = 0.dp)