This is very true. I currently have two of each in my Elcy city which will all be sold now that I've been accepted into an excellent FS there.
I'm waiting to do so until the FA starts only to keep myself from using the space for other buildings, because I haven't planned out my FA layout in Elcy yet.
I need to sell the pair in Sinya as well - they're more than outdated anyway.
__________________________________________________
I think this might be the crux of the issue. When I first got mine, I compared them to the manus for the goods they produced for me as well and was pleased. But when I take that space and use it for another manu of my boosted goods, I can trade for the other two I'm not boosted in that the Traveling Merchant produces.
Unlike other event buildings, this one doesn't give culture. So when you're in an active FS or neighborhood with an active trader, they really do become space wasters. But if you happen to be in a dead zone, they may be essential. (They were for me, several times.)
As for the non-boosted manus, you won't need them for your T1+1 good until you get to Sentient Goods. Right now, they not only eat up your space, they're also eating up your gold and supplies when you manufacture goods with them rather than trading for them. I'm not sure whether the manus or the wholesaler is cheaper as I never sat and did the math on that to find out if someone in a true trading dead space would find them essential for that reason.