This is a bug in the _load_pretrained_model() function of transformers/modeling_utils.py when loading sharded weight files. The state_dict is applied to the empty model per shard. This is problematic as the quantized weight and its meta data(*.quant_state.bitsandbytes__nf4) may be stored in the different shards. The quick-and-dirty fix is to merge tensors from all shards into one state_dict. Similar issues have been reported on the unsloth github issue 638