I'm not going to be able to add to much more than what's already been said. I'm fairly similar to you, except I called myself "heteroflexible" for a really long time. Now, having been with a transman or two, I'm also realizing that pansexual is probably a more appropriate label for me. (Well it is if I want to advertise and attract. ) Any way, I haven't been in a romantic relationship with a man, but I have met a few men in my life that I've had freakish crushes on. And my "freakish" I only mean that there was nothing done to bring them about. One guy I can remember working with in the late 90's, when I was in my late 20's. There was just something about him. He was tall, funny, and nice and when he looked at me, I had to do everything not to show that I was weak in the knees. He was definitely straight and West Indian -- a population that sometimes has serious issues with homosexuality or anything like it -- so I never dared let on. I also didn't fully understand it, but we didn't work together for long anyway. So, what I'm saying is sometimes if the right person comes along...who knows? But you sound like you've had more than your fill of fun anyways.

I'd estimate (from personal experience) that at least 90% of bisexual males have no romantic interest in other males. Sex? Yes. Partnership, companionship, love? No way. I'm sure a lot of it is to do with closets, that bisexuality is often derided by homosexuals and heterosexuals alike, and the fact that bisexuals have the option of maintaining a facade of heterosexuality. One that's made more credible by having a visible partner, the binary opposite sex to themselves.

Sexuality and romantic interest are two distinct things. Add in a third for gender interest, given that 'man' does not necessarily correlate to 'anatomically male', nor 'woman' to 'anatomically female' these days. Sexual interest is at least three distinct things: It isn't just a spectrum, it's several of them. The way I currently see it:
Asexual, bisexual, heterosexual, homosexual: the sexes, in relation to your own sex, of people you would be willing to have sex with, all other preferences being met. In 99% of situations, this boils down to who has a penis and who has a vagina.
Gay, pan, straight: the gender presentation/identity, in relation to your own gender presentation/identity of the people you'd be willing to have sex with, all other preferences being met.
Aromantic, biromantic, heteroromantic, homoromantic: the sexes (in relation to your own) that you'd be willing to consider a relationship with.
Gayromantic, panromantic, straightromantic: the gender identities/presentations (in relation to your own) that you'd be willing to consider a relationship with.

So, for example, a man with a penis who is only sexually interested in women with vaginas is both straight and heterosexual. A man with a penis who is only attracted to women, but doesn't care whether they have a penis or a vagina, is both straight and bisexual. Romantic interest follows a similar pattern.

This is how I currently make sense of the different labels and the different spectra involved in sexual interest and romantic compatibility. It is not my intention to dictate anyone else's labels for themselves here.

It is all a spectrum. There is no wrong way to do sex or relationships. But many people think there is. Some people are only comfortable with having one person to love and have sex with. Some can only love one person, but can have sex with many people. Some can love many people, but only have sex of they also love each person they have sex with. Some people can only love and have sex with one person, but are ok with the person they love having additional sex outside that relationship. The whole idea behind poly, in my opinion, is that there is no cheating. Each person, and each relationship is discussed and open. Each person needs to be honest with themselves about what they are comfortable with and can accept for themselves and their partner(s). If you cannot be ok with a particular choice, you may have to move on until you find a situation that you are comfortable with. I think the whole benefit to polyamory is we are ooen and honest in what we want and respect other enough to never lie or cheat. But of course it hurts to think you might have found an amazing person who can't accept some portion of your relationship or sexual goals, and one of you may need to move on to other relationships. And with so many possible combinations, it can be hard. But hopefully with patience, we can all find honest happy relationships that meet all our sexual and emotional needs. And, like you said, some people will never believe that you can have sex just for pleasure, that you must also be in love, or not love them if you can have sex with someone else. That is just who they are, and in that case I would say you just need to find someone who can understand, and be ok with, who you are and what you want. Not easy, but worth it.

I’m also bi but never enjoyed such a relationship. You were lucky to have that. I keep hearing about how wild women are but never experienced that. I didn’t tell my wife about my sexuality and it broke us up after 17 years (she found a gay porn site that I went to). I hope you are able to find it again but I now find that I like being alone. Rarely do I engage in sex at all (except with myself). Good luck to you in finding it again.