Most culture event buildings (as long as that is what you are referring to) should be replaced as you gain things that are more efficient on a per-square basis. If I am looking at the right city, you have a Sorcerers Ship at chapter 2 and you are now in chapter 6. You probably have something that is better, so I would replace it.
Unless you really just like the look of it!