Did John McClane know Bill Clay was really Hans Gruber? If so, how?

Those two scenes go like this:

(Hans jumps down from an inspection and ends up crouched at John's feet. With a gun pointing at his head, he pretends to be an hostage until John calms him down..)
John: What the fuck are you doing up here? What were you looking for?
Hans: I was ..trying to get up, on the roof and.. see if I could signal for help. You know.. (looks back to where he'd stashed the gun) It's just through here. Why don't you come and help.
John: Hold it. Forget the roof.
(as Hans goes to pass through the passage between industrial units to retrieve the gun. John swings the weapon around so it is again pointing at Hans)
John: I said forget the roof, they got people all over it.
Come on, you want to stay alive, stay with me.
(Hans 'has a look' before scene breaks to FBI)
(Cuts back. John takes out cigarette pack with 2 left)
John: You smoke?
Hans: (smiles) ..Yeah. Thanks.
(John has one cig. in mouth as he passes the pack and remaining cigarette to Hans. There is a directory board over Hans left shoulder, showing names and locations of employees. John has has weapon pointed in Hans' general direction. Not directly at him, but not far off either. Hans takes the last cigarette and slips the empty pack in his pocket as John pulls out a Zippo.)
Hans: You don't work for Nakatomi. And you're not one of them..
(John, now turned further away from Hans, smiles wryly)
John: I'm a cop from New York.
Hans: (Hans shows some surprise) ..from New York! (chuckles)
John: Yeah.. Got invited to the Christmas party by mistake. Who knew?
(Hans chuckles, notices John's bare feet, John starts chuckling..)
John: Ehh.. Better than being caught with your pants down, huh?
(Hans also chuckles, John glances at directory)
John: I'm John McClane. You're uh..
Hans: (Hans looks blank for a moment) Clay. (smiles) Bill.. Clay.
(John glances back at Directory and sees a Clay, W.M. from the 29th floor. Nods to himself.)
John: Know how to use a hand gun, Bill?
Hans: I spent a weekend at a combat ranch. That game with the guns that shoot red paint. Probably seems stupid to you.
(John has been doing things with the hand gun, seems to put a clip in it and cock it.)
John: Nope. (John holds out gun, to Hans.)
Time for the real thing, Bill.
(Hans looks at the gun, looks at John, then slowly, reaches out to take it.)
John: All you got to do is pull the trigger. Come on. (Walks past Hans.)
(Hans drops his cigarette, stamps it out, and speaks into his radio..)
Hans: 33. Stockwerk. Kommt sofort.
(He takes his time, aiming the gun at John as John walks slowly away, with a 'I knew it..' look on his face)
'click' (as he ..again, cocks the weapon)
(John takes out his cigarette and slowly turns back to face Hans.)
Hans: Put down the gun, and give me my detonators.
(John takes a drag on his cigarette)
John: Well, well, well.. Hans.
Hans: Put it down. Now.
John: That's pretty tricky with that accent. You oughta' be on fuckin' TV with that accent. (Slowly walks back towards Hans) But what do you want with the detonators Hans? I already used all of the explosives. ..Or did I?
Hans: I'm going to count to three.
John: Yeah. Like you did with Takagi?
(Hans pulls trigger) 'click'..
John: Oops.. ('click, click, click)
No bullets. What do you think - I'm fuckin' stupid, Hans?
('bing' as elevator arrives)
Hans: (dripping with sarcasm) You were saying?

Transcribing those two scenes as I was watching them again, I think that John was not fooled for a moment, but was trying to draw out as much information as he could from this chance hostage. I don't think he at first realized it was Hans himself, but John knew all along that it was one of the gang members. He was probably surprised that the leader would be forced to do his own 'trivial tasks' away from their main stronghold on the 30th floor.

Note that most of the early time he has a gun pointed in Hans' general direction. But he reaches a stage where he realizes he must convince 'Bill' that he himself is convinced of his story, so gives him the (unloaded) weapon.

When John glanced at the directory and found the name, he was probably thinking something along the lines of:

OK - this guy is no dummy - better be real careful with him.

Allowing Hans to get on the radio indicates he was not quite careful enough, he'd gambled a bit too much in order to get more information, and it almost cost his life (for the umpteenth time that evening).

If someone can find it, there is actually an official answer to this question out there.

Approximately 10-15 years ago, I was watching Die Hard on TV. I'm not sure which channel it was on, but my gut instinct is that it was a Turner channel (TBS or TNT). Anyway, whichever channel it was had shot a few brief interviews that they would air clips from during commercial breaks. After this scene aired, they played a clip which specifically addressed this question. The clip was essentially the scene from the movie with voice over narration.

Whoever was narrating the clip (I believe it was director John McTiernan, but I can't swear to it) claimed that in the original script, John was able to recognize Hans because everyone in the group wore the same watch. The narrator then claimed that he "decided to take that bit out and allow people to draw their own conclusions about how John figured it out."

Confirmed (and elaborated upon) here by the screenwriter, Steven E. de Sousza:

"Originally, [when Gruber and his men] get off the truck, the camera craned up, you saw them in a circle and Alan Rickman says, 'Synchronize your watches,'" de Souza explained.

"They all put their arms out in a circle with the camera moving down and they all had the same Tag Heuer watch. If you notice, the first guy Bruce kills almost by accident going down the steps, he searches the body, looks at the IDs."

Also in a longer cut of the scene, McClane begins to notice the terrorists' watches. "He steals the cigarettes, which is a laugh," de Souza continued. "He looks at the watch, which gets another laugh because you're thinking he might steal the watch. As he kills each guy, he notices they all had the same watch."

"When he talks to Dwayne Robinson, he says, 'I think these guys are professionals. Their IDs are too good. There's no labels on their clothes and they all have the same watch.'"...

"When Bruce [Willis] offers the cigarette to Alan Rickman, Bruce sees the watch. You see his eyes look at the watch. That's how he knows that he is one of the terrorists."....

However, the "synchronize your watches" bit was cut in the end because it came into conflict with the plan for the terrorists to escape in an ambulance, which was added towards the end of shooting.

Argyle (De'voreaux White) the limo driver crashes into the escape ambulance, but that ambulance was absent in the "synchronize" shot, thus leaving an even bigger potential gap.

De Souza added: "[Director] John [McTiernan] says to the editor, 'Get the scissors in there. Cut as soon as you can when they get off the truck so we don't see there's no ambulance.' Now without 'synchronize your watches,' all of these moments where Bruce looks at these guys' watches makes no sense."
- Esquire, The Screenwriter of Die Hard Finally Explained a Major Plot Hole; March 2, 2017

I have a feeling that the fact that Clay" said he was an American smoker, He blew his cover by not questioning the odd brand of cigarettes. If he was American, why wouldn't he be like "wtf kind of smokes are these?". John must have known once "Clay" puffed on it without question, he must of been a European terrorist.

I'd say that John was simply being cautious.

What are the chances of some civilian other than him also escaping for so long, and climbing around instead of hiding? Also, Hans was quite hesitant to give his name when asked.

A stretch, but when John walked into the building, didn't the reception guard say that the 30th floor was the only floor with people still? Clay bill was listed on 29th. If I am right, I have no idea how I noticed this...

This scene has always puzzled me, so thanks to everyone for providing input. My take was the way Hans answered when John asked his name. Typically, people would answer with their first name first, then last name if need be. Like "my name is Bill... Bill Clay" or something. But never "my name is Clay... Bill... Clay" the way Hans answered. The directory board showed last names first, then first names. So my thinking is that Hans had examined the directory previously, noticed the name "CLAY, WM" and immediately answered with Clay first, the paused to understand that "WM" was William, but shortened it to Bill.

I'm surprised to see a question with so many answers, none of which I think are correct.

John: That's pretty tricky with that accent. You oughta' be on fuckin' TV with that accent. (Slowly walks back towards Hans) But what do you want with the detonators Hans? I already used all of the explosives. ..Or did I?

Hans: I'm going to count to three.

John: Yeah. Like you did with Takagi?

(Hans pulls trigger) 'click'..

John: Oops.. No bullets. What do you think - I'm fuckin' stupid, Hans?

It's no coincidence that John reveals to Hans he was watching his conversation with Takagi at the same time it's revealed to the audience that he gave him a gun with no bullets. The implication I immediately got while watching the film was that John recognized Hans from that encounter and was toying with him from the start.

We see from John's perspective that he sees what Hans is wearing before the shooting. That might be enough to go on for a trained detective.

But more importantly, AFTER the shooting Hans stands up from out of his chair, where judging by what the audience saw previously, John should be able to see at least the back of his head, and perhaps his profile.

I think the simplest answer is that John saw Hans then, a fact that was hidden from the audience to preserve the suspense later in the film.

What I think happened was Hans saw the same board moments before. When McClain asks for his name, Hans thinks for a moment and thinks of one of the names on the board he just saw. Luckily John glanced at the same board too and probably thought, what are the chances, that out of the hundreds if not thousands of people who work in the building, that he actually was one of the few named on the board. Also, I'm not sure, but he also might've been tipped off because the directory says Clay works on the 29th floor but the party was being held where all their offices were located - on the 30th floor. Otherwise why would they put that part in the movie at all? If he believed him it wouldn't matter if he said his name was Ron McDonald, not everybody in the building was listed on the board. He could've even pretended he was a guest at the party and made up any name. I think Hans was trying to be slick but McClain was too sharp.

It's the cigarettes. The first time John talks to the LA police officer and is describing everything the terrorists have (guns, bombs, etc) he states that they are European and he makes a very obvious pause before he's remarks about their cigarettes.

if there was no John McClane, Hans would eventually have called police. Perhaps he was going to use the name of one of the staff (William/Bill Clay) or one of his cronies would do it. They would say "my name is bill clay, send the cops quick...terrorists here etc etc" hence when John asks him his name, he knew one. Maybe Bill Clay was their inside man who gave them layout of building etc. (long bow) If you look closely in film pending your tv, most of the floors have a floor directory near elevator.

But yes, the cigarettes are a give away. Any America would comment on the smokes and say "gee these are crap/great" or even cough. The fact he took up the smoke so well with no comments was the give away for John

He held the cigarettte like a European (thumb and finger), not like an American (between two fingers).

As far as the "How" part of the question, remember that John had at least heard Hans' voice and perhaps even seen part of his face and haircut prior to their meeting. Early on in the movie, McClane kills a a terrorist, steals his gun, writes "Now I have a machinegun. Ho ho ho." on the dead man's sweatshirt, and sends the body down in an elevator to the floor where the hostages are being held. Hans and another guy enter the elevator car to investigate and they spend a minute talking, not knowing that McClane is mere feet away, on top of the elevator car, listening to them. You can also see some of the light from the elevator shining on McClane's face, so it's possible he saw Hans' hair, the suit he's wearing, maybe even his face.

He somehow knew even before that. "I got invited to the Christmas party by mistake. Who knew?" Why not say that his wife invited him unless he was trying to protect her from Hans knowing that his wife was was there? Almost like he already knew that if the terrorists found out his wife was there they could use her as leverage. He had also heard Hans voice previously, accent or not. He is a trained police officer, to survive in an ordeal like that, it would be nothing to be observant enough to know that something was off.

The offering of the cigarette IS THE MOMENT. Europeans grab smokes and smoke them like joints. This was in the book/screenplay. The horrible accent in the movie was IMPECCABLE in the written version. Willis offered the smokes and immediately knew that he was one of the terrorists simply by the way he smoked it. Google "Hans Gruber smoking". He's hitting it like a joint.

I used to wonder this myself, but seeing it recently it's obvious. The questioning and murder of Takagi is witnessed by McClane (in hiding) and Grubber is present, so it is quite possible he was spotted by McClane prior to their later meet.