Awesome thanks Froyok, that video cleared up the worst of the problem. My normals were already in order, it was just down to misaligned UV islands with the pixels of the lightmap.
On those building pieces I am currently using a lightmap scale of 128 on the roof and 256 on the wall. I've tried increasing the resolution, even up to 1024 (
) but it doesn't seem to change anything, the blobs just get sharper edges.