Poked at this, made it work by setting DYNAMIC_DRAW for the buffer usage and using MAP_WRITE | MAP_READ instead of READ_WRITE for the mapping