TL;DR

There is no difference in the phasers, but in the situations. The one time Khan gets hit by surprise, the other time he sees the attack coming.

Long answer

When Kirk storms the bridge of the USS Vengeance together with Khan and Scotty, they take take out the whole crew except for the Commander. Kirk then gives Scotty a sign by nodding at him, after which Scotty fires a phaser burst at the unsuspecting Khan, taking him completely by surprise.

Khan is out cold. Is he really, though?

That is exactly what Kirk and Scotty planned prior to the attack:

KIRK: The minute we get to the bridge, drop him.
SCOTTY: What? Stun him? Khan? I thought he was helping us!
KIRK: I am pretty sure we are helping him.

Khan is not prepared for this surprise attack from the people who he thought to be working with him.

I believe that is the main difference to the other incident you mention, the fight on the floating barge:

While Uhura teleports in behind Khan, he still becomes aware of her presence in time to turn around and engage her.

He sees her pointing the phaser at him and is able to physically and mentally prepare for the attack. This is enough for the genetically improved Khan to withstand multiple phaser shots without being knocked down. It still has a severe effect on him and allows Spock to finally overpower him.

(Screenshots and Transcript taken directly from the movie)

There was no difference. Khan played possum while he figured out Kirk's plan or the best course of action. As a genius chess master type, he had figured Kirk isn't a complete idiot, and knew being double crossed was a possibility. He faked being stunned and waited until the right moment to turn Marcus's head into a crushed soda can.

During the Spock fight, there was no more reason to fake it. If he did, he would be allowing them to capture him.

That, and his physical, super human strength is higher than the typical stun setting of the phaser would be set for. Like in real life, some people can just shrug off normal tazers.

Besides the element of surprise, I think adrenaline has a lot to do with it. Not only was he in a knock-down drag-out fight with Spock at the time he shrugs off the phaser shots, he was also possessed with wrath.