So I ended up figuring this out, although I'm not sure if it was the best solution or not.

I had to reinstall WASMedge using an updated install script and specify the path to that .so directly in the install script (LIBCUDART_PATH). That made it recognize CUDA and the GPU, and then I had to mess with context size for memory reasons, and then it was up and running.