It turns out it was no fault of our code, non-reproducible, some bad luck.

Index-level quotas for a couple of indexes were reset to 0GB. It is important to note that quota for an index is by default 10GB, and it is not possible to for a user to set it to 0GB.

Google Cloud support was helpful. After an investigation by Product Engineering Team, the explanation was that "during a recent upgrade of our quota infrastructure, there was a transient error that may have temporarily reset your quota bucket.".

If this happens to you, writes on the index are suddenly and effectively disabled. Quota increase request take days to be processed. No workaround. Therefore, your best bet may be to create a new index, copy the documents over to the new index and begin using the new index.

After your request for quota increase has been granted, you may re-use the index. However, we noticed possible signs of data corruption, so it might be safer to delete all documents, delete the index, and start the index anew if need be.