You can cut down trees and build a house/cabin. The mechanics for a settlement counting as a hamlet, town, village etc would be a little different. You'll essentially be playing the survival elements a bit more.
Check this out for more infos: https://chroniclesofelyria.com/blog/6549/DJ-18-Kingdom-Land-Management
And while things are mostly destructible, punching trees won't chop down a tree, you need siege weapons to break down walls etc. Logical things.