Yep, dumping all the dead cities will drop your fellowship from 22 members to 11 or 12. And Arendyll is tough to find new recruits, so once you are down to just active cities, it may be easier to find another fellowship of about the same size and merge with it. Either all your people join them or they all join you. If you do the joining, that can also get you out of being AM, if it is getting too stressful.