I have figured it out. Since I have not found any tutorials on this I will leave some hints for people working on a similar problem.

In my case the problem was the rp_filter. Packages are dropped if the source ip is not from reachable from the interface the package arrives from. Since Host IP is not reachable by sending the package to a NIC the packages where dropped.

Another pitfall to consider is connection tracking. If you change the source and ip, the connection tracking entry might be invalid which can lead to dropped packets as well