Awesome! I was afraid I was going to have to restart.
I'm just going to change the difficulty before fighting him rather than when I'm just about to kill him.
I did miss this one audio diary in Rapture Central Control, but good thing I have another save file with it. Looks like I don't need to restart after all.