I found how to do it using the internal method ipysheet
ipysheet.easy._assign(sheet.cells[my_cell_idx], new_cell_value)