Found the answer - or a possible answer:

Instead of the expensive

Buffer.concat([allData, data]);

This is much faster:

1.) Pre-alloc a Buffer:

const allData = Buffer.alloc(mbExpected * 1024 * 1024); 
// if the size is too small, node seems to auto-expands the buffer

// store the current position in the buffer
let currentPosition = 0;

2.) Fill the data into the buffer:

allData.fill(data, currentPosition, currentPosition + data.length);
currentPosition += data.length;

With this, the code runs a lot faster!