After you set the visibility to GONE, add

ConstraintLayout.LayoutParams layoutParams = (ConstraintLayout.LayoutParams) tvQuestion.getLayoutParams();
layoutParams.topToBottom = R.id.tvQuestionNumber;
tvQuestion.setLayoutParams(layoutParams);