Reports

What I observed is that res.body.Read(buf) reads only a small number of bytes. ... Why is this the case?

The io.Reader documentation explains why:

Read reads up to len(p) bytes into p. ... If some data is available but not len(p) bytes, Read conventionally returns what is available instead of waiting for more.

The call to res.Body.Read(buf) returns what is available instead of waiting for len(buf) bytes of data to arrive from the server.

Use io.ReadFull to wait for len(buf) bytes of data.

n, err := io.ReadFull(res.Body, buf)

This is not your question, but the code in the question does not handle the return values from res.Body.Read(buf) correctly. Specifically, the code does not handle the case where data and an error is returned. Here's an attempt to improve the code.

            n, err := res.Body.Read(buf)
            if n > 0 {
                // Handle any returned data before handling errors.
                if !yield(bytes.NewReader(buf[:n]), nil) {
                   return
                }
            }
            if err == io.EOF {
                // Reached end of stream.  Done!
                return
            }
            if err != nil {
                // Something bad happened.  Yield the error and done!
                yield(nil, fmt.Errorf("unable to read: %w", err))
                return
            }

A better solution is to io.Copy(part, rest.Body) where part is wherever the application is writing the data.

79243188