It depends if the galaxies and stuff move around in the universe, like cars do on the road. I don't know if they do or not, and I have a feeling neither do the Scientists.
RD has a point, you are just thinking of the universe as a static thing that shouldn't be moving.