Tested it out, same behaviour. Ok the UI artifacts thing ist just a byproduct of the actual problem. It seems that it is really a issue with the garbage collector (speaking now in dev-language). To reproduce:
create a artboard file,
create a couple of artboards with a decent amount of textures,
look at the increasing Ram usage.
Now remove some of those artboards and look again at the Ram usage.
For me nothing happens, still at the same high amount.
If you would save now and exit and open the file again (restarting AD), the Ram usage would be lower than before. So for me this means that garbage in the memory isnt thrown away :). Btw same counts for opening additional files in the same session (closing would not clean up my memory).