In the constructor, use

setHasStableIds(true);

this will generate unique identifiers for each view which will not change. This may result in slightly higher resource usage but make the recycler view items stable