With whom is Felicity Smoak in love, exactly?

With Oliver, whose rejection was the reason she sought love elsewhere.

Well, first of all, it is not that unusual for people to be in love with more than one person in their life. In addition to that, the show doesn't really run short of romances, which is an important aspect to its story, especially the love interests of Oliver Queen. While season one concentrated on his feelings for Laurel and the conflicts arising from his guilt and her emerging feelings for Tommy, the second season put him into a much more light-hearted relationship with Sara based on "shared interests", and the 3rd season now goes the whole tragic "I can't love her because that puts her in danger" route with Felicity, not to mention all the occasional flings (from Helena to Isabel) and his love for Shado in the past. So there is an apparent endeavour by the show's writers to make human, and especially romantic, relationships an important side aspect of the show.

But let's talk about Felicity in particular and the development of her feelings throughout the show. Felicity more or less started out pretty much as Arrow's "hacker assistant" and with her slightly cute nerdy behaviour she became more or less the little sister to Arrow that Thea is to Oliver.¹ Season 2 also introduces Barry Allen as an apparent love interest to Felicity, yet again based on quite similar character traits and interests. But due to unforeseen consequences there isn't made much more out of this until season 3 (more about this later).

But it isn't until the very last episodes of season 2 that there is an actual romantic angle suggested between Felicity and Oliver (apart from her occasional drooling over his shirtless fitness exercises), even if he only uses that as a ploy for Slade Wilson at first when he tells her he loves her in order for her to play the bait for Slade. So it isn't clear to me if the prominent Felicity-Oliver romance from season 3 was already planned by the writers at the end of season 2 or if they just came up with that for season 3 based on this scene, especially seeing how very surprising that angle was at least to me. And in this season there is the classic conflict between the hero and his tragic love that he cannot live for the risk of putting his loved one in danger (interestingly the exact same problem he used as advantage at the end of season 2). And the two are played pretty much as the perfect loves of their lives to each other throughout this season.²This tragic romance drives itself through the entire 3rd season and hinders a successful and fruitful romance between Felicity and Oliver up until the finale, when Oliver finally puts down his costume and starts to live.

And in fact this conflict is also the reason why Felicity has quite a few attempts at other romantic relationships, and why those ultimately can't succeed. As mentioned, she fits quite well with Barry and their relationship is flaeshed out a little more during her appearances on the sister show The Flash. But at the end of the day, she as well as Barry have to admit that they would both only be substitutes for their real loves -- since Barry also has that tragic "I never told her and now it's too late" thing going with Iris. Refer to the ending of The Flash episode "Going Rogue" (S01E04):

Felicity: What is wrong with us? We are perfectly perfect for each other.
Barry: Yet we're sitting here, pining for people we can't have. I guess what they say is true, opposites do attract.

And from a writing standpoint it is also a natural decision, since while a romance across the two shows and more frequent appearances of the characters would be interesting, it is logisitically more feasible to keep the two worlds and their romances rather closed inside the individual shows.

And this same subsitute love relation also goes for her relationship with Ray Palmer. She has again found someone whom she shares quite some interests and traits with and who won't push her back (and maybe she also starts to recognize some aspects of Oliver in Ray's endeavour for avenging his late wife and becoming a hero). And for some time she really enjoys that relationship and it brings her happiness. Yet at the end she and Ray again have to realize that Ray is ultimately just a substitute for the real love interest she cannot reach. Refer to Arrow episode "The Fallen" (S03E20):

Ray: Last night was a bit of an eye-opener. Insofar as realizing something I should have realized a long time ago can be considered eye-opening. Last night, when you were saying good-bye to Roy, you and Oliver...You still have feelings for him. When I told you that I loved you at the hospital, you responded with "Jell-O". Seeing you two together last night, I realized you don't love me because you're in love with him.

So at the bottom line, while Felicity had quite a few romantic relationships with other characters, the reasons for those and for their failure were ultimately grounded in the fact that she couldn't live the one true love she really desired, which is definitely Oliver Queen (or so does the show tell us with its season-spanning tragic romance arc). So maybe Barry is right with his above statement and the fact that Felicity and Oliver don't seem to fit that much together actually is what makes them predestined for each other.

1) _{Which is also notable since in the beginning the Arrow-Oliver difference was quite a bit more distinct, before pretty much everyone knew his identity and became badass heroes themselves.}

2) _{Interestingly it is also season 3 when the actual distinction between the Oliver-family and the Arrow-family starts to completely dissolve, with the rich playboy Oliver Queen pretty much disappearing and the rest of his remaining family being dragged into the Arrow-world (ref. footnote 1).}