I wouldn't go there if I were you

Every settlement has something wrong with it, some reason why people would rather stay away.  It might not be enough to keep the town completely isolated (if they find gold, people show up no matter how bad it is) but every inhabited place would be better off if something were different.

Poor Village, Ast Ralf

Choose two reasons why folks stay away from this place.  The first reason has led to the second, in some fashion.  (Feel free to tone these down or ramp them up as needed.)

Why do folks stay away? (d10)
1	under threat of attack
2	hard to get to, difficult terrain
3	dirt poor, nothing to offer
4	history of disease
5	violent and aggressive towards outsiders
6	swindlers and thieves prey on travelers
7	crop/mine/fishing has nearly run out
8	strange cult-like devotion
9	bad/dangerous things live around here
10	inwardly-focused, no care for the outside world

Joseph Smith, CCA Christensen

Decide who (or what) folks are most afraid of around here.  This might not be their enemy, just someone whose power they fear.

Who/what do they fear most? (d20)
1-5	their rivals or competitors
6-7	the authorities, government, or the warlord who rules over these parts
8-9	a poorly-understood group of people on the other side of a river/mountain/swamp
10	far-off invaders rumored to exist
11	each other
12-13	starvation, running out of necessities
14-15	creatures that may or may not exist
16-17	the days to come, according to their understanding
18-20	whatever their neighbors/allies fear

The Battle of Sitka, Louis Glanzman

Let's try rolling up a few.

First we have a human town, some corn farmers who live in a valley.  They've been attacked many times, so they've become violent and aggressive towards outsiders.  They (along with the other humans in the area) are afraid of creatures rumored to exist in the surrounding woods.

Next is an elven mining settlement.  This place has a history of unexplained disease, which has led the people to turn to a strange religious movement.  Their cult-like devotion and rituals are strange enough that people don't like coming here anymore.  They're afraid of their competitors in the mining business, who would probably use underhanded tactics to drive them out.

Now let's look at a scattered group of giants who gather once a year.  They're very poor, desperately eking out an existence in the high country.  As a result, they never have anything to trade, and have become inwardly-focused, not interested in the goings-on of the larger world.  Starvation is a constant source of fear.

Here's a family group of goblins living in the deep woods.  They're very inwardly-focused, refusing to interact with anyone else.  As a result, they've ended up without medicines goblins normally get in trade, so they're always suffering from ailments.  As suspicious as they are of outsiders, they're even more afraid of each other, constantly on the watch for treachery and attempts to seize power by their kin.

Lastly, let's consider one of the surviving elven cities.  They're known as a den of thieves, a place you should never go if you value your coin more than your life.  Pirates sail from their harbor and raid up and down the coast.  As a result, they're always picking up plagues and diseases from one port or another.  They're afraid of creatures that come out of the sea and drown sailors.  These creatures may not exist, but they certainly believe in them.

---

Roll up some trouble for your own settlement:

stay away from trouble
first problem
which has led to
what they fear most