It works when you specify plot.background in the theme inside patchworkGrob():

grid.arrange(
  patchworkGrob(pA + 
                  plot_annotation(tag_levels = list(c('A')) )
  ),
  patchworkGrob( plot_spacer() + pD +
                   plot_layout(widths=c(-0.35, 1)) + 
                   plot_annotation(tag_levels = list(c('D'))) & 
                   theme(plot.background = element_rect(fill = "transparent", color="#808080"))
  ),
  nrow=1
)

My package versions:
ggplot2_3.5.2
gridExtra_2.3
patchwork_1.3.0
ggpubr_0.6.0