By the time Jefferson was old enough to have his remarks remembered, he was operating in the mainstream of society, among the movers and doers of his day. The late eighteenth century was not the time to make wild fashions statements just to "express oneself", the way we're more free to do these days.
Even today, you'll notice that the movers and doers of today's government wear well-made suits and ties, solid white shirts, and polished shoes when they're doing official work. Their leisure clothes follow a rather narrow range of styles as well. It's called "establishment dressing".
Movie stars are expected to dress in rather flamboyant styles. Popular musicians wear flashy, sexy costumes. Rock stars have long hair and casual dark clothes. Nine out of ten students seem to be wearing jeans, and either athletic shoes or flip-flops. Evidently most people dress themselves according to expectations, and get unfavorably noticed if they vary too far from their particular norm.
There are other ways to express one's individuality besides dressing oddly, although many people go through that stage for a time.