Simply kill everyone in the towns, this is harder than it sounds. First you must break down the doors or destroy every breakable building, often townspeople can get stuck inside if you just break the door down so destroy the building. However in each area there are also gates that connot be passed until certain parts of the game.
In Nordberg there is an area which can only be accessed when you have green minions to destroy a barrier and another that requires blue minions to move a raft. In Everlight an audience later in the game will give you a sidequest to open the final area there which contains the last 20 townspeople.
If you cannot find a villager try gating out of the map and then back to it, this will spawn all townspeople again and replace the doors on houses that still contain any.