As a non-American, I’m very confused by this. If it’s a town, it’s not rural by definition. Because, you-know, it’s urban.
Also, could we get a definition of town vs small town. Do you not have the concept of a village? (Village in the UK would be a settlement with a population of a couple of thousand, with usually a pub, local shop, maybe a post office and primary school if you’re lucky).
We use the word differently. In the past I think we used it more as you do, because “going to town” had the connotations of going to a big city.
“Town” in American usage can mean anything from a small urban center (like under 10k people) to an incorporated municipality that has only a post office and tons of farms around it.
Basically we don’t say “village” here. So town is the smallest word we have. But it has a big range.