Is it easy to lost a title? That highly depends on how much work you do to secure it.
- Recruit people to help you secure your position as Count.
- Get to know your neighbors before the game start. Build a relationship. So that they won't try to invade you.
If you lose your title, you will lose your lands and manor. You may or may not keep your pets/items/etc, depending on whether the invader that took your title also decides to raid your house and take all of your stuff.
Everyone is taking risks by being a noble. Although there are undoubtedly many advantages as well. Just try to minimize the risk by taking actions to secure your position.