The word "heathen" is old -- dating from before the 12th century AD.
The word -- signifying to us an irreligious or unconverted person -- is probably derived from the English word "heath," meaning "an extensive area of rather level open uncultivated land usually with poor coarse soil."
So at some point in our past, at least in England, the heathen were likely country folks, those who lived out in the heaths.
The same is true with "pagan": "from Late Latin paganus, from Latin, civilian, country dweller, from pagus country district." The "pagani" were the hayseeds from out in the sticks.
We in America think of cities (especially very large ones) as being hotbeds of irreligion and vice, and the countryside as the dwelling place of strong piety. Not so to our ancestors, who knew that the last places where the old pagan religions resided was in the countryside.
Culture flows from urban centers outward. That's why St. Paul kept going to cities to preach the gospel: that's where the influence was. Jerusalem, Athens, and finally Rome, where he was martyred.