Thanks to @brebs and @TessellatingHeckler for your responses. Both of them worked as intended.
I took influence form @brebs solution and implemented an accumulator to basically count the number of occurrences over the pass and unify it with Count in the base case:
fourExactly(X, List) :-
count_occurrences(X, List, 0, Count),
Count == 4.
count_occurrences(_, [], Count, Count).
count_occurrences(X, [X|Tail], Acc, Count) :-
NewAcc is Acc + 1,
count_occurrences(X, Tail, NewAcc, Count).
count_occurrences(X, [Y|Tail], Acc, Count) :-
X \= Y,
count_occurrences(X, Tail, Acc, Count).