As Benny says, perhaps you have different DLC downloaded. Or you both have NAT issues. Are you able to connect to him in other multiplayer games? Are you able to join an xbox live party with him?
One good test would be for both people to power off xbox, power off the internet router, wait 30 seconds, turn on the internet router, then once your internet is back up, turn on the xbox and try again.
Sometimes your router gets hung up. I've had connectivity issues in the past and restarting everything on both ends fixes it most times.