I ended up figuring this out and will post my solution for others having this problem.

It seems like MediaElement works differently with MAUI .NET 9.0 and above.

I had to do two more steps in order to make the embedded video play.

1. Update the LaunchSettings.json file and change the "commandName" value to "MsixPackage"

2. Remove the following line from my .csproj file:

   <WindowsPackageType>None</WindowsPackageType>

After making those changes the embedded videos would play without issue.