Yes, you will lose the runes that were put into the circle. Any runes that have not yet been inserted into a ring will still be yours.
(Edit: it is one of the tricks people use to get more rune shards for a wonder they want. Build one you don't want then destroy it so you can fill the ring again, allowing you to get more broken shards for other wonders.)