Here is the vercel.json that fixed my issues. Mainly pointing vercel builds to server.ts, not server.js.

{
  "version": 2,
  "builds": [
    {
      "src": "src/server.ts",
      "use": "@vercel/node"
    }
  ],
  "routes": [
    {
      "src": "/(.*)",
      "dest": "src/server.ts"
    },
    {
      "src": "/(.*)",
      "dest": "dist/server.js"
    }
  ]
}