Often enough, a lot of us end up with the "horse we had to have and fell in love with" or "one that we had such a strong connection to" or one that is absolutely stunning.
We then find out that we are not a fit at all - he's too jumpy and spooky, not good on the trails, terrible in the arena, doesn't pay attention, too fast, too slow, no finished maneuvers, runs away from us, hates being saddled, and on and on.
We may be a beginner who now just has way "too much horse" or an experienced rider, even trainer, who just doesn't feel like the taking the time it takes to get the darling where we want him - we just want to be able to sit and go pretty much with only a few minor tuneups.
It's important to take action and do something about the situation. It will be the best thing you can do for yourself and a real gift to your horse. It's nothing to feel bad about - though it is not an easy thing do do because of the affection you have developed.
But it needs to be done - for both of you.