Short answer, no. It is possible to get along with only one armory. However, there is at one point a quest that requires you to have three armories at a certain level or higher, and having multiple armories increases the number of troops you can train at once. While it doesn't increase the training speed, what that does mean is that you can set a bunch of slots to train and not have to worry about checking in for 3 or 4 hours because of how long it will take to finish training all of them, instead of having to check in every, say, 30 minutes because of how few troops you can train at once.
In summary, yes, you will probably appreciate having at least 3 armories.