After lots of digging, trying all kinds of AIs, asking friends and blaming my image loading code, I finally found out that I could just set the Garbage Collector to be as agressive as possible:

GC.Collect(2, GCCollectionMode.Aggressive, true, true);

Why?

Since Images are usually somewhat big, they are being put into the Large Object Heap.

Garbage collection usually tries to leave some LOH-Memory allocated to use it as a cache for future objects (as far as I know, feel free to correct me!), but in my case, this was not necessary, and by setting the GC to agressive, I could get rid of all the wasted memory and drastically reduce RAM usage!

It took me a really long time to figure this out, and I am really glad that I found out.

If you are having simmilar problems with large piles of unused memory, try making the GC collect the different generations in agressive mode. Even if it's not images but byte arrays in general.