Who did send three gunmen to kill Harmonica in the beginning if Frank didn't know Harmonica?

Frank knew that Harmonica was coming. Because Harmonica told Frank he will come.

• Harmonica asks the three guys "where's Frank?"
• Then he asks if they have brought a horse for him.

This shows that he announced himself prior to coming. Probably even saying on which train he will come. Frank was expecting him (perhaps not knowing who he really was, though.)

Frank is aware of Harmonica's existence and (later) his pursuit to kill him. He is just curious why. A better constructed, reversed "it was Tuesday" trope.

You see, not knowing who his opponent was and what were his motives left Frank without leverage. He mocked people, laughed at their weakness. Having this mysterious person who only says names of people Frank killed leaves him open as dead people don't have soft spots.

Harmonica had contacted in advance a man named Wobbles (interpreted by Marco Zuanelli) to arrange a meeting with Frank:

[Harmonica] You know, Wobbles... I'm kinda mad at you. Frank wasn't there.
[Wobbles] Ow!
[Harmonica] He sent three friends.
[Wobbles] I don't know nothing, I swear. I only arranged the meeting the way... the way you wanted it. I don't know why Frank wasn't there.

Given Frank did not know Harmonica, he probably found the meeting suspicious and decided to send the three gunmen to neutralize the potential threat.

Harmonica has obviously been chasing Frank for a while, hence the trap at the start of the movie.

Frank is merely wondering why Harmonica is so eager to kill him.