I disagree. The Quidditch matches weren't all that difficult, and I was able to 5-star most of them on the first try. The same thing goes for the Potion and Duelling Clubs. The game is very simple once you get the hang of it.
I had lots of fun with the game, and I never got mad at it. And believe you me, I'm easily frustrated by games. Personally, I think you might just suck at flying.