Scratch that, apparently this only works some of the time. The rest of the time hammer just does has it pleases.
I actually found a wall that had black spots on the inside and outside faces. The whole deleting and getting everything back in fixed one of the sides, but not the other.
It's really frustrating, if someone has any idea I'd love to hear it.
EDIT: And other times it actually creates completely new black spots. It's just completely random, I can't find any pattern to what is happening.