What are the characters thinking/trying to convey during the scene where Cheyenne first meets Harmonica?

It's a powerplay between the two. Cheyenne can tell Harmonica is not your average guy - he's not flustered when an escaped criminal walks in. So Cheyenne is trying to provoke him to test him to see if he can back up the tough-guy act.

So he:

1. Takes his weapon. No response.
2. Challenges him to use it. No dice.
3. Ok, he's staying cool so far. He smiles out of respect.
4. Insults him a bit. Harmonica still doesn't bite.

Now Cheyenne has some idea of what Harmonica is like, and a bunch of respect for him.

The scene in question can be found here:

First of all, I think you are missing a whole lot of steps before that.

-4. Cheyenne enters the bar. Everybody is tense and afraid of him. People stop eating, the barkeeper tries to grab his hatchet. It is completely clear who is in charge and Cheyenne only needs as much as look at them to achieve dominance. Strings are playing, Cheyennes theme is played in intervals.

-3. Silence while Cheyenne starts drinking.

-2. A harmonica starts playing in the moment where Cheyenne lowered his defenses to drink. If it had been a shot, he would be dead now. An insult, a defiance. But where does it come from? Cheyenne squints his eyes. Actually everybody is surprised where it might be coming from.

-1. Cheyenne locates the musician in a dark corner, makes an angry face and pushes the lamp over to his corner illuminating him. Now the Ball is in harmonicas field.

0. Harmonica simply takes up eye contact and holds it. Cheyenne moves towards him, trying to intimidate him. Harmonica does nothing, not even smile and still continues with his harmonica. Cheyenne musters him and notices bullet wholes in his jacket. Since he is still fairly healthy, it is likely to assume that the jacket is from a dead person. And it is also likely to assume that Harmonica killed that person. Cheyenne smiles for meeting another killer. Harmonica does not smile back but makes a strange movement with his mouth.

1. Cheyenne does not like that reaction at all. Breaks eye contact for the first time (losing this stand-off as well). He then looks down and notices the gun that has been laying within arms reach the entire time. He is shocked upon another insult that the other guy had a gun ready the entire time. This calls for a duel to regain his pride and to regain control. But harmonica does not react noticeably. He starts to become more tense as the gun is pulled away from him, but for some reason he still seems in control of the situation. Most likely he knows that Cheyenne has some kind of thieves Honor that forces him to hold his duels honorably.

2. Cheyenne pushes the gun towards him. To force him to that very duel. He orders him to pick up the gun. First Harmonica keeps playing. But Cheyenne stresses his order once more. Now the duel is definitely on. If he picks up the gun, bullets will fly. So he does as he is told. He picks the gun up, but in a funny way, while still using one hand to play his harmonica. This is of course not a fair duel, so Cheyenne will not continue with the duel. But it is also another act of disobedience. Furthermore it takes a lot of guts to go into a duel with an upside down pistol.

3. Cheyenne smiles. The smile has many layers. In the first second of his smile, he breaks eye contact and looks down. He loses yet another duel. To a guy holding an upside down pistol without firing a single shot. He smiles out of relief, amusement and desperation for multiple reasons:

• The other guy is no - immediate - danger to him
• the other guy seems interesting and - in a way - a stray dog like him
• he has not felt that way before
• it is the first time he has been humored in a long time
• he cant beat the other guy by his usual methods

3.1 He picks up on the joke and simply owns the third guy who is grabbing for his gun by pointing at him with his fingers. It might have been his kind of humor as well, he just hasn't been on the receiving end of it yet.

3.2 When Cheyenne approaches Harmonica, Harmonica smiles.

3.3 Cheyenne regains a bit of dominance. First he uses the third guy to burst his chains. Then his gang arrives.

4 When Cheyenne is trying to leave while he is ahead, harmonica walks back and asks for his gun back. Now even insulting him in front of his gang.

4.1 Cheyenne tries not to enter into another competition and sends one of his gang members forward to handover the pistol

4.2 Harmonica reinforces the insult by claiming to have seen 3 dead gang members (and who would have put the bullets into them?).

4.3 Cheyenne claims that his gang members don't die. And makes an enumeration with 2 entries.

4.4 Harmonica refers back to the music he played before (music can also be seen as a duel (see bancho duel), because there you need to react in tune to what the other person is doing. Cheyenne was trying to keep harmonicas pace before but was limping behind) and makes fun of that very short list of reasons. Yes, it is sarcasm. It depends a bit on how you see their relationship on whether it is really a big insult or just two "bros" having a go at each other.

4.5 Cheyenne threatens him to a gun fight again.

4.6 Harmonica simply picks up his instrument and turns around forcing Cheyenne to shoot him in the back.

4.7 Cheyenne tries to have the last word of the conversation and tells him to not play any bad tunes

4.8 When he is just about to exit (again), Harmonica plays a series of bad notes, putting the last insult into Cheyennes boot.

4.9 Cheyenne leaves.