Yep, that's the worst part of it, for certain. Here is an alternative method I find useful. Instead of finding individuals I scan through the fellowships looking for ones that are small. I then look to see if there members are active and if there are only one or two still active and the others aren't I try to recruit the two. There is a caveat to this:
If the AM is active I contact him/her first to insure I'm not poaching. They may not reply, but it's just common courtesy, I think. If not, I contact the members individually but let them know if we have room for more than one.
In the long run I find this method better as people in a dead or dying fellowship often want to remain active and just don't move due to not knowing anybody other than those in their fellowship. Once they move they usually become good members.
AJ