Story-related, unmissable. Note there is a bug people sometimes encounter with this that makes the duel at the end of this quest line unwinnable (you can't hit him and he can't hit you). There are two workarounds for this:
Reload a save from the beginning of the Isle of Thieves, making sure to follow Slayne the whole way and not go exploring on your own until after the cutscene with the chest.
During the duel, either lure your opponent out onto the fields where there are monsters and stand next to your enemy so the monsters hit him while they're trying to hit you, or attack the townsfolk and do the same thing, so they're the ones attacking him. Make sure to have plenty of healing items. Once your opponent is defeated, you can simply run away from town until they stop chasing you (everything should be ok once you have the option to fast-travel); when you return, the townsfolk will no longer be hostile.