you can display the size of the font by pixel

add this line to the toolbar

    //[{ header: [1, 2, 3, 4, 5, 6, false] }],
<ReactQuill
        theme="snow"
        value={""}
        modules={{
          toolbar: [[{ header: [1, 2, 3, 4, 5, 6, false] }]],
        }}
      />

it will display like this:

if you want to display pixel size instead of 'Heading'

add this CSS code to global CSS

.ql-snow .ql-picker.ql-header .ql-picker-item[data-value="1"]::before {
  content: '32px' !important;
 }

 .ql-snow .ql-picker.ql-header .ql-picker-item[data-value="2"]::before {
  content: '24px' !important;
 }

 .ql-snow .ql-picker.ql-header .ql-picker-item[data-value="3"]::before {
  content: '18px' !important;
 }

 .ql-snow .ql-picker.ql-header .ql-picker-item[data-value="4"]::before {
  content: '14px' !important;
 }

 .ql-snow .ql-picker.ql-header .ql-picker-item[data-value="5"]::before {
  content: '10px' !important;
 }

.ql-snow .ql-picker.ql-header .ql-picker-item[data-value="6"]::before {
  content: '8px' !important;
 }