I'm very late to this question, but we find VSTO add-ins work fine with both 32-bit and 64-bit Office when using either x86 or AnyCPU. Using x64 only works with 64-bit Office.

We see some small performance improvements working with the 64-bit except when running our tests from Nunit. Our nunit tests with 64-bit Office are much slower than before. Not sure why that is but at least they do still work.