How do I upgrade Neovim to version 0.9+ on Ubuntu 20.04 (GCP terminal)?

You can either download the prebuilt binaries from release page or build neovim from source.

Official Guide

What’s the easiest and most stable way to set up Neovim with full IDE-like functionality (ideally without installing a full GUI)?

Can be a debatable question. I honestly think v0.11+ native Neovim can be enough. It already ships with most of those features you are looking for so using native Neovim could actually be the most stable way (here is my blog post.)

kickstart.nvim is pretty good configuration template to start learning about fundamentals (which will lead you to stable IDE-like functionality at the end)

If you want ready-to-go solution to save your time, there are many Neovim config distros like LazyVim (not lazy.nvim, that's different thing), NVChad and AstroNvim. Choose what you prefer and don't forget to pin all dependencies to prevent breaking changes.

Is NvChad or SpaceVim suitable for cloud SSH workflows?

They all run on TUI (neovim) anyways so doesn't really matter.

Are there best practices for performance or config when working fully over gcloud compute ssh (e.g., color issues, clipboard)?

Many people use different methods. So here are some:

• Usually developers upload their neovim config to remote repository and clone on machines they work on.
• Some people don't want to download their config in remote server, so they often use solutions like sshfs or distant.nvim.
• For color-rendering issues, usually using modern terminal emulator with proper $TERM environment variable and using 'termguicolors' Neovim option would fix it.
• For clipboard, Neovim supports osc52, so you can copy/paste through the ssh connection (see :help clipboard-osc52)