Making granite is free. Until you get the quest to sell the portal, you could keep making granite for no charge, and then you can use that to upgrade some of your roads to Dwarven style roads later, if you want to and haven't already, but for that to work, you also need copper, coin and supplies. So if your roads are already upgraded to dwarven, or you don't plan to, then the only possible use for the mines is to store materials you might need later. Your Main Hall holds limited coins and supplies, so by keeping the mines for a bit, you have an extra storage of those you can use when needed by selling them off. If your Main Hall is usually near empty, and you don't want to convert roads to dwarven style, then once you get into fairies there is no reason not to get rid of the mines.