The problem with a term like gentleman is that it has evolved in meaning over centuries from someone who could live independently from an income derived from landholding in early modern Europe to evolving codes of socially perceptive and considerate behaviour.
Whee! I was wondering how long it would take for this to come up. I've known good and decent men who would take great offense to being labeled as gentlemen. To them, that word means someone who is rich, a bit of a dandy, who has never done an honest day's work in his life. To others, it may simply mean a polite person who is of decent and moral character. The word gentleman is just one of those words which, as you said, has evolved over the centuries, and has many different connotations.

There were certainly some "gentlemen" in past centuries who may have been rich but were absolutely lacking in morals or courteous behaviour. Just as there are some "gentlemen" today who are dirt-poor but are honest, chivalrous, and upstanding. And what's interesting is that the definition of a gentleman doesn't seem to necessarily be split by nationality (unlike many of the other words we often find have completely different meanings between the UK and USA). It seems to have different definitions across society, even amongst people of the same social station. 'Twould make for a very interesting sociological study, I think!

One might even make the case that it's tied to the cultural sense of honour, which is equally ambiguous across social strata. Whilst one man may consider himself a gentleman, defending his honour, by engaging in a fist-fight or a duel over a slight verbal insult, another man may consider it most ungentlemanly and dishonourable to do so. And these fellows could even live in the same town. It just depends on how they were raised, what they believe, and how they view the world.