I think a really big + to unarmed combat is how reliable it is when it comes to unexpected combat. We know always wearing armor isn't practical in CoE just the same way always carrying a weapon wont be practical, although you may get away with short swords, daggers and weapons of that nature.
There will also most likely be places were weapons aren't allowed, may that be in a city or inn etc, and I doubt a farmer will carry around a sword and shield when they're trying to tend to their crops, and I imagine this will be the case for many people in general not just farmers.
With all this in mind I think unarmed combat is seriously underestimated. Although I wouldn't argue unarmed is a good way to fight someone in full armor with a sword and shield, I would 100% argue unarmed can be effective versus someone dressed in cloth or even leather wielding a dagger, which I'd imagine the typical bandit or robber lurking about would use.
I don't think unarmed combat will ever reign supreme when it comes to a anticipated fight or battle, but when it comes to living out your life day to day and dealing with the unexpected it may be the best secondary combat skill to know ... because you'll always have your hands.