Hazeltown is just a pop/culture hybrid. And it is decent, but I don't think it is the best building of this type by any means. Pop/culture hybrids are offered in every event as daily prizes. I would not use a Teleport Building spell on 1. I'd just wait to win something better and replace it.
edit: I just re-read your original post. What I was talking about was replacing 1 pop/culture hybrid with a newer one. What you are talking about is replacing Hazeltown with a gems manufactory. You would need a huge surplus of population to make this switch work, because Hazeltown is providing pop, while the gems manufactory is going to use pop. When you factor both of these in, the swing in population is going to be considerable.