I don't think it actually hurts anything, other than slows down progress that requires population (need more than already would for specific thing to first compensate the negative). I've actually done this on purpose. I don't recall now what building it was or why I chose to sell it, but I was aware it would remove more population than I currently had. I let population be negative for a bit (not long term at all) and just factored that difference in the population gain I was next pursuing. I think I was making room for something I thought was valuable in other ways and needed the space immediately (unwilling to wait for next expansion). It might be controversial whether it should be done on purpose (or to what degree, for what justification), but I wouldn't sweat it. Moving forward, know you now know something you didn't before and you can avoid it or choose it, either way knowing the consequences.