After the room of butterflies disappears, travel down the path until you find the town of Passwall and talk to Shelden. He will tell you that before you can actually enter the Shivering Isles proper, you’ll have to pass the Gatekeeper. There’s no way around killing the Gatekeeper because the key to the gates is hidden inside its body. If you’re daring or powerful, you can try to face the Gatekeeper directly, but he’s quite a challenge. There are two side quests that you can undertake that will make him much easier: Gatekeeper Arrows and Tears of Relmyna.
Gatekeeper Arrows – Find Jayred Ice-Veins and he will tell you that he can make arrows from the bones of former gatekeepers, which will provide extra damage to the Gatekeeper. Travel to the Gardens of Flesh and Bone and pick the lock to enter. You’ll have to fight off a number of Shambles before Jayred can collect the bone chips. Wait several hours and talk to him again when he will give you twenty Gatekeeper Arrows.
Relmyna’s Tears – Find Nanette Don, the apprentice of Relmyna Verenim, and raise her disposition, if necessary, until she reveals that the tears of Relmyna, the creator of the Gatekeeper, seem to damage it. She’ll also mention that Relmyna visits the Gatekeeper every night at midnight. Visit the Gates of Madness at midnight, taking care to not be noticed (a high sneak skill is very handy here). Wait until you receive a message telling you that Relmyna has dropped a handkerchief. Find the handkerchief and you’ll have three bottles of “Relmyna’s Tears”.
One of the easiest ways to handle the actual battle with the Gatekeeper is to poison your Gatekeeper Arrows with Relmyna’s Tears and attack the Gatekeeper from afar. If you would prefer to use a melee weapon, you can still poison it with Relmyna’s Tears. Once the Gatekeeper is dead, take both the “Key to Mania” and the “Key to Dementia” off of his corpse. Enter the Shivering Isles via whichever gate you prefer and travel to Sheogorath’s Palace in the city of New Sheoth to speak to Sheogorath and finish the quest.