(Unable to add a comment right now due to low rep)
But have you tried loading within a custom scope?
custom_objects = {"CustomLayer": CustomLayer, "custom_fn": custom_fn}
with keras.saving.custom_object_scope(custom_objects):
reconstructed_model = keras.models.load_model("custom_model.keras")