Did Oberyn "The Viper" ever expect to win the trial of combat?

In the books Oberyn Martell was infamous for always poisoning his weapons. It's part of the reason why he got the nickname "The Red Viper". Lacing his blade with poison accomplished two things:

• A potent enough poison can disable an opponent if the duel lasted long enough.

• Ensures that the opponent dies even if he survives the duel (wins or surrenders).

There is no POV chapter from Oberyn in the book so we do not truly know his chain of thought. However, it is highly likely that he did expect to win the duel.

He is an experienced and accomplished fighter and, although the TV episode has the fight more one sided than in the book, he is correctly confident he can beat the Mountain. He poisoned his blade (hence he tells Tyrion not to touch it) but he is known for doing that.

Most likely (as I said we cannot know for sure) was that he intended beating the Mountain into the ground and forcing him to confess before leaving him too badly injured to continue. Then he would be able to walk away having forced Gregor to publicly implicate Tywin for ordering the rape and murder of his sister whilst Gregor suffered a slow, agonising death.

If Gregor was unable to even sit up I doubt anyone would question that Oberyn had won the fight but if there was an issue then Oberyn would still have had the option of killing him there and then.

On the other hand if Oberyn lost then, providing he wounded Gregor, he would still have his revenge against him even if Tywin had not been implicated.

Knowing that Oberyn's prime motive was to avenge his sister, he would make sure that The Mountain would suffer even if Oberyn himself lost. Given his strong desire for vengeance, he may not have even volunteered as Tyrion's champion if the Crown's champion were not Gregor Clegane. Sure, he felt a bit sorry for Tyrion when he recounted his experience of meeting an infant Tyrion, but if you weighed Oberyn's empathy for Tyrion against his desire for vengeance, you'd find that revenge greatly outweighs empathy.

Unfortunately, in the end, Oberyn was too wrapped up in trying to exact his revenge that he lowered his guard just long enough for The Mountain to kill him. If Oberyn weren't clouded by his desire for revenge (hence a normal 1v1 combat), Oberyn could have bested The Mountain. Look at his style of fighting: he's light on his feet, agile, and knows The Mountain's weaknesses (Gregor Clegane suffers from bad migraines and is almost always on Milk of the Poppy, and Oberyn's constant shouting and taunting disorient him). If Oberyn were not so emotional during the fight, he would have kept the upper hand easily. For a similar battle, remember Bronn vs. Ser Vardis: Bronn was light while Vardis was heavily armored. Bronn knew he couldn't win head-on, so he tired out Ser Vardis until he had the upper hand. Oberyn was fighting the same sort of battle, except his emotions clouded his better judgment.

Now, whether the poison actually finished off The Mountain or Qyburn's mad science has created a Fraken-Gregor, we'll have to wait and see.

Manticore venom is said in the book to cause instant death once it reaches the heart but can be tampered to be more slow acting and painful. My guess is that he wanted the poison to torture Gregor so he can talk even with the wounds.

In the show the poison seems to be painful, so instead of holding a sword over him he just yells as he probably expects the poison to be so painful that Gregor begs for death and confesses (that also add the bragging of being the guy who made the toughest in the kingdom cry for death).

Oberyn is supposedly considered an impressive warrior and is renowned throughout the Seven Kingdoms for his skill hence his nickname 'The Red Viper.' Oberyn will no doubt be aware of Ser Gregor's own martial skill especially following the reports of the rape and murder of Elia Martell and her two children (Yes I'm aware of the possibility surrounding those events in A Dance with Dragons). The Dornish themselves are especially hateful of the Lannisters who they believe to be behind Elia's death. Also remember Oberyn and Elia are brother and sister hence why Oberyn goes after the Mountain in the first place.

I think there is something in 'A Feast For Crows' in a conversation with Oberyn's paramour Ellaria Sand in which she states that Oberyn knew he did not have to kill Ser Gregor simply cut him for the poison to work its magic so he may have believed there was a chance the Mountain would best him and certainly most of the people watching were shocked to see Oberyn doing as well as he was but I definitely think that Oberyn was aware of his chances of survival. While I can't provide any information to back up my own opinion, have a read through this as it will give you more insight into the character and will go further in depth over some of the things I've mentioned, hope I've helped: http://awoiaf.westeros.org/index.php/Oberyn_Martell