Long story short: I think it's highly likely to be a wise choice to use other methods for formatting a text file for styling, as @AD7six suggested in the comment section.

As @j6t stated in the comment section, I feel like this issue has to do with the metadata. I'm not sure whether this is relevant or not, I found that git uses stat values.

Anyway, I ended up thinking that it should be impossible to use this program as a clean filter. The reason is that if I made the program not to modify a file being processed, then the index would differ with the worktree, and if I had it to modify, then git would recognize the file modified. Double trouble! This is a dilemma; either way, I will be in trouble.

Nevertheless I just wanted to make use of it since I already had spent two weeks to make this "simple" program. Perhaps it's my mistake to write it in C. So I considered the following alternatives:

1. write a script which wraps git add. In other words, the script executes my program first, lets it modify the worktree, and then runs git add -- <file>....
2. write a pre-commit hook that executes my program to have it modify the worktree and runs git add to update the index.
3. somehow make the program executed whenever I save a file.

However, #1 would require me to distinguish which is an option to git add and which is a file. I don't think argument parsing is an easy task. So I skipped #1. Meanwhile, #2 seemed promising but it has a flaw in that git add would occur twice: the first add by the user and the second by the hook. Also, actually another problem remains: how to handle the partial stages, i.e. git add -p? In this case I can't update the index with the fixed file in the worktree since doing so would nullify the partial stage. Maybe it would be possible to manage to fix and update the index only somehow without touching the worktree but doing so seems stressful. In the case of #3, I don't know how.

In the end I had no choice but to choose the #2 but I tried hard to avoid the handling of the partial-stage case by executing my program if and only if a file in the index has a modified state or newly added state, and the file does not have local changes, i.e. the index equals to the worktree.

In order to do so, I parsed the output of git status --short because its first and second columns represent the state of the index and the worktree, respectively.

fp = fopen("tempfile", "w+");
system("git status --short > tempfile");

while (readline(fp, &line)) {
   if (line[0] != 'A' && line[0] != 'M')
      continue;
   if (line[1] != ' ')
      continue;
   ...
}