Since the introduction of teleport spells I simply no longer believe this argument.
Teleporting buildings only gets you past 1/2 of the challenge.
Yes, you can avoid the "sliding puzzle" part of trying to move pieces, but teleporting doesn't help at all with putting pieces back together in the most efficient way.
As I mentioned earlier, I believe there is a reason that our residences and/or workshops change direction every chapter- it's so that we continually re-design our cities. Armories too.
If buildings didn't go from 7x3 to 4x6 it would take almost no effort to find upgrade space: Just move the road 1 square to the left, upgrade, rotate, done.
Basically, if we could rotate at will, you might as well have all 5x5 buildings.