Ok so the model is working fine now with high accuracy as much as 96% with further improvement if trained for more epochs and very low loss too as low as 0.15. This is achieved by following steps:

1. Freezing initial layers about 80-90% for e.g, in EfficientNetB5 I froze first 400 layers and made the last 20 layers trainable.
2. Reducing the additional layers in Sequential model and using only 4 additional layers:

layers.GlobalAveragePooling2D(), layers.Dense(1024, activation='relu'), layers.Dropout(0.5), layers.Dense(len(class_labels), activation='softmax')

3. Increasing the initial learning_rate to 0.0005 and number of epochs to 50.
4. Increased the number of images in the dataset by combining 2 dataset from kaggle as the initial has less images.

By following these steps although model started with low accuracy in initial epochs it gradually reached satisfactory accuracy and this time the loss was also low from the start approx. 1 as compared to 100(absurdly high) previously.