Correctly track annotations in a HeapGraph