This achievement won't unlock if anything else in the game killed the boss. The most common example is with the beaver boss that almost always kills itself on the spikes in the cave. If this was how the beaver died, your achievement won't unlock. Because of this, I recommend to kill each boss at least twice. Just walk forward for the environment to change and then the boss that you just found should spawn again.