After testing your code on CPU and GPU, I found some interesting bit. On CPU, I get the following error:

tensorflow.python.framework.errors_impl.InvalidArgumentError: {{function_node _wrapped__SparseSoftmaxCrossEntropyWithLogits_device/job:localhost/replica:0/task:0/device:CPU:0}} Received a label value of 2 which is outside the valid range of [0, 1). Label values: 2 1 1 0 [Op:SparseSoftmaxCrossEntropyWithLogits] name:

Interestingly, this error does not come with GPU. There, the nan values appear instead.

It seems that on CPU it is explicitly checked if the logits vector for one prediction is long enough to make all labels possibile. Let me show you an example with modified code:

target_class_ids = tf.Variable(np.array([[2, 3, 1, 0]]), dtype=np.int32)
pred_class_logits = tf.Variable(np.array([[
    [-0.0738682151,  -0.0052,  0.78],
     [-0.0405795932, -0.0215,  0.32],
     [-1.68359947,   -0.54,   -5.6],
     [-2.13260722,   -0.111,   0.45]]], dtype=np.float32))
loss = tf.nn.sparse_softmax_cross_entropy_with_logits(
      labels=target_class_ids, logits=pred_class_logits)
print(loss)
tf.Tensor([[0.63222516        nan 0.2814241  3.081086  ]], shape=(1, 4), dtype=float32)

CPU output (important part):

Received a label value of 3 which is outside the valid range of [0, 3). Label values: 2 3 1 0 [Op:SparseSoftmaxCrossEntropyWithLogits] name:

Here, I modified the labels a bit and made the logits vector longer of each prediction.
Notice the nan value at the index of the 3 label? Because with 3 logits per prediction, only 3 labels (0, 1, 2) are possible as output. On CPU, this is explicitly told. "3" as a label is just not possible with only a logits vector of length 3.
In your example, the logits vector for each prediction was only of length 1. So every label that is bigger than "0" goes to nan on GPU, and throws the error on (my) CPU.

I checked the linked repo, and it had a config with NUM_CLASSES=1 as default. Did you override it to the correct number of classes in your case?