Before you can start these challenges you need to head to the Manzanita Post, you can buy the 20 Tomahawks for $10 each. Once you have the Tomahawk you can proceed to complete the following challenges:
Tomahawk Mastery Rank 2
Kill one enemy with the tomahawk used as a throwing weapon, one as a melee weapon, and one from horseback.
You can do these three just about anywhere that you encounter enemies. Animal enemies do not count towards the criteria, but you can head to any hideout and complete these in a matter of minutes.
The melee attack doesn't haven't to be done on an enemy, it seems to count if you equip a bandana and kill any of the none hostile characters around town. Head to Thieves' Landing and take out any of people walking around. Just walk up to the person you want to kill, stand close and use the tomahawk as if you were to throw it, instead you will kill the person you're facing by hitting them with the Tomahawk instead.
With the next two, which involve throwing the tomahawk, head to Pike's Basin, use dead eye and carefully pick off one of the guys hiding behind the rocks at the start. Once you complete that part of the challenge, jump on your horse and do the same.
Just remember, using deadeye may help you aim, but once you throw the enemy may duck down or move. So aim carefully.
Tomahawk Mastery Rank 3
Kill a flying bird with a tomahawk without using deadeye.
This sounds fairly difficult, but it is fairly simple and easy to do. Head to any of the hideouts, I decided to stick with Twin Rocks as it's small and quick. Kill everyone and complete the hideout, once you complete the hideout get onto the roof of the house. You will now have several vultures flying above you, these are one of the larger birds in the game so they are easy to hit without a struggle. Load up on tomahawks and just carefully aim and throw.
Tomahawk Mastery Rank 4
Kill 5 enemies in a row without missing with the tomahawk.
This is another challenge, which can easily be completed at Twin Rocks. Take out the first two guys high up on the rocks with a rifle, they are easy to miss so there is no reason to waste tomahawks on them.
You will have four guys left in front of you, then another three will exit the house once they are killed. Carefully work your way around the cover and use your deadeye and melee enemies with the tomahawk as much as possible. Doing this will allow you to score the kills without missing. You will also perform an instant kill rather than having to aim carefully with your throws.
Tomahawk Mastery Rank 5
Complete a hideout using only tomahawks.
This final challenge can also be completed at Twin Rocks. You can repeat the same as Rank 4, but make sure you aim carefully for the guys up high on the rocks. Try to get as close as you can and use deadeye as much as possible.
Just take your time and use the tomahawk's melee attack often so you don't waste how many you have.
It seems that once you earn Rank 4 it resets the counter for the hideouts. If you complete Master Rank 4 then continue inside the house and kill the last enemy, it seems to count for the last challenge. If this doesn't work for you, it might have just been a freak glitch, but it's worth a try as you won't lose out on anything.