Most men don’t do it theses days but the main reason is to show themselves as proper or strong. It just makes them seem manly. They want to seem like James Bond (007) with his gentlemen acts that make him a kind and tough man at the same time. But like I said, nobody does that these days and they wear beanies these days, not hats.
Removing one’s hat has been a sign of respect since medieval times. It began by removing the hat in church to show respect for God.