Why did Lord Tywin try to execute Tyrion?

He didn't. His original plan was for Tyrion to live - but Tyrion blew it after Shae's testimony.

Remember this dialogue from S04E06 when a concerned Jaime is trying to reason with Tywin (adapted from this article):

JAIME: I’ll leave the Kingsguard. I’ll take my place as your son and heir, if you let Tyrion live.

TYWIN: [no hesitation] Done!

JAIME: [looks flummoxed] ...

TYWIN: When the testimony is concluded, and the guilty verdict rendered, Tyrion will be given the chance to speak. He’ll plead for mercy. I’ll allow him to join the Night’s Watch. In three days time, he’ll depart for Castle Black and live out his days at the wall. You will remove your white cloak immediately. You will leave King’s Landing to assume your rightful place at Casterly Rock. You will marry a suitable woman, and father children named Lannister. And you will never turn your back on your family again.

JAIME: [looks flummoxed] ...

This plan - which Tywin had clearly been working on for a while - was win-win-win for Tywin:

• His favoured son becomes heir again
• The family conflict is resolved honourably
• Justice is seen to be done - and the reputation that you don't mess with Lannisters is sustained by the swift justice
• No more Lannisters die - and the reputation that Lannisters are untouchable is sustained by the fact Tyrion is still alive despite being found overwhelmingly guilty of king-murder
• Everyone is where Tywin wants them to be:
  • Cersei returns to raising Lannister-in-all-but-surname royals,
  • Tyrion is far enough away to be no risk of embarrassment or further conflicts with Cersei,
  • Jaime continues the family name and lordship, far enough from Cersei to squash rumours.

Or, it would have been if Tyrion hadn't freaked out during the trial after Shae betrayed him, made that speech, deliberately ruined Tywin's plan and demanded a trial by combat... Once Tywin's work to save Tyrion had been rejected, he wasn't going to bail him out a second time. This Lannister had paid his debt.

Why did the Shae incident happen? Short answer, Tywin seemed to be trying to disgrace Tyrion, teaching him another lesson not to be indiscreet with whores against Tywin's orders. This then backfired because Tywin underestimated how emotionally invested Tyrion had been in his relationship with Shae. For more on that see Why this turn in Shae's character? and Why did Tywin allow Shae to be a witness at the trial?

This isn't necessarily the full story however, there may be more to come. GRRM the author of the original books has said mysteriously: (quoted in that first linked answer)

I won't comment on the Tyrion / Tywin issue. Perhaps future volumes will throw more light on it.

One important unresolved question: Tywin seemed to know that Tyrion was innocent from his (lack of) reaction when Tyrion confronted him:

TYRION: I am your son and you sentenced me to die. You knew I didn't poison Joffrey, but you sentenced me all the same. Why?

TYWIN: Enough. We'll go back to my chambers and speak with some dignity.

Tywin's the kind of realpolitik player who doesn't consider the fact his less-favoured son is innocent any kind of obstacle to using the perception of his guilt to further his own interests - but it's interesting how little he seemed to do to investigate Joffrey's actual killer. Did he believe Tyrion was the killer? Did he know he was innocent? Or did he (perhaps most likely) not care what was true and just acted on what was prudent?

But that's a whole different question (as is the question of, if he had lived long enough to secretly investigate and find Joffrey's real killer, would he prosecute them or thank them?).

Tyrion is accused for the murder of Joffrey.

Every witness that came accuse Tyrion

Meryn Trant recalls when Tyrion slapped Joffrey and reminded him the fate of the Mad King. That is considered as an implicit threat.

Cersei recalls the famous Tyrion's line: "I will hurt you for this. A day will come when you think you are safe and happy, and suddenly your joy will turn to ashes in your mouth. And you will know the debt is paid." Again, that is considered as threat.

Varys heard Tyrion warn Joffrey that "just now kings are dying like flies". Again a threat.

Shae lied, but gave specific and credible details because she was Tyrion's secret lover. So for the court, Shae was reliable.

Trial by combat

A trial by combat means

If the accused party is victorious, they are cleared of all charges. If the accused or their champion is defeated, however, then they are considered guilty and condemned to death.

Oberyn Martell was Tyrion's champion and Gregor Glegane was Cersei, the accuser's champion.

As Oberyn died, Tyrion was sentenced to death, so technically, Tywin has not personally decided to kill Tyrion.

I don't think he tried/wanted to execute him, but he simply didn't have a reason to help him.

He obliviously could've helped him (He was the "main judge" of the trail and choose the other two judges, one of them is Mace Tyrell who will vote the same as him) but why would he help him at all? He sees him as an embarrassment to the Lannisters name and there's a chance that he actually killed Joffrey. So he simply let the justice takes it course!

Also, remember that when Jaime confronted him, Tywin denied any involvement in the manipulation that happened in the court (he had no reason to lie) and quickly accepted the deal Jaime offered to free Tyrion.

I think there are a lot of really good answers here that make fantastic arguments to Tywin's line of thinking in this situation. Honestly, I'm not sure if we can ever really know what he truly wanted or rather what was more of a priority for Tywin's interests, considering how he handled the Shae situation, letting her testify against Tyrion and because, there may not have been a lot he could of done anyways outside of removing Shae from the situation (and for me, that does seem like icing on the cake, showing Tywin's true feelings)...

However, I also think there is evidence to suggest that Tywin wasn't particularly interested in saving Tyrion's life even before this situation arose.

In fact during season one after Tyrion escapes from the Eyrie & picking up Bronn as his Champion; rallies the hill tribes to his will by making lavish promises to them, then bringing them before Tywin to fulfill these promises (which ALL were to keep Tyrion alive, by the way). Their conversation starts with Tywin bringing up a rumor of Tyrion's demise...

1.08 The Pointy End - Transcript:

TYRION: Uncle. Father.

[TYRION stands before his father. BRONN, SHAGGA, and two other tribesmen, TIMETT and CHELLA, join him.]

TYWIN: The rumors of your demise were unfounded.

[TYRION looks irked.]

TYRION: Sorry to disappoint you....

Then in the next episode, Tywin seemingly punishes Tyrion for everything Tyrion has put Tywin through and commands Tyrion to lead his hill tribe (and presumably Bronn) in the front of the vanguard!

1.09 Baelor - Transcript:

TYWIN: I do hope your savages are going to be of some use; otherwise we've wasted good steel on them.

KEVAN: The great hairy one insisted he must have two battleaxes. Heavy black steel, double-sided.

TYRION: Shagga likes axes.

ANTHOR DAYNE: That's putting it mildly......

[TYRION turns to face ANTHOR, looking surprised to see him.]

TYRION: Ah, the great Anthor Dayne, the twin brother to the Sword of the Morning. You're a long way from home, Ser; what brings you to the Riverlands?

[ANTHOR raises an eyebrow at TYRION.]

TYWIN: Enough. When the battle commences, you and your wildlings will be in the vanguard.

[TYRION looks surprised and a little apprehensive.]

TYRION: The vanguard? Me and the tribesmen, on the front lines?

[TYWIN nods softly, and has a slight grin on his face. TYRION still looks apprehensive.]

KEVAN: They do seem rather ferocious.....

ANTHOR DAYNE: You have any experience in combat, Lord Tyrion?

TYRION: [annoyed, to ANTHOR] Perhaps you'd be kind enough to join me. [to KEVAN] Ferocious is putting it mildly. A Moon Brother stabbed a Stone Crow over a sausage. Three Stone Crows seized the Moon Brother and opened his throat. Bronn manage to keep Shagga from chopping off the dead man's cock, which was fortunate, but even still, Ulf is demanding blood money, which Shagga and Gunthor refuse to pay.

TYWIN: [sternly, but amused at the same time] When soldiers lack discipline, the fault lies with their commander.

TYRION: [frustrated] Surely there are ways to have me killed that would be less detrimental to the war effort!

TYWIN: There'll be no more discussion on the matter.

Now obviously Tyrion never makes it to front lines, but even seeing how easily he was knocked out and trampled upon, along with Tyrion often acknowledging he is not a good fighter, which is showcased in previous events, making deals and getting people to fight for him, it does imply that Tywin knew there was good chance Tyrion wouldn't make it...

1.10 Fire and Blood - Transcript:

TYWIN LANNISTER: They have my son!

In another scene in the next episode, Fire and Blood, viewers see Tywin get rather distressed when Tywin learns that his other [more treasured] son Jaime was captured and learning how Robb Stark laid a trap. I only included this bit to show Tywin's bias, because he never reacted that way toward's Tyrion.

He's also sends Tryion to King's landing to fill in for Hand of King in his absence, which he knows will irk Tyrion and Cersei alike, but not without threatening or warning him not to take his prostitute [Shae] with him. Almost implying that this was his last chance...

Tywin may know that Tyrion is not his son but the result of a union between Tywin's wife Joanna and Aerys Targaryen II. Tyrion serves as a daily reminder to Tywin of his wife's infidelity which would explain why he hates the very sight of him. It would also explain why dragons Rhaegal and Viserion do not kill Tyrion but seems to "know" him. In Tywin's view his two "real" Lannister children are flawed, Jamie because he lacks the intellect and interest to become a man of power and Cersei because she is a woman. Without actually stating it, Tywin occasionally advises Jaime but knows his line is doomed to mediocrity or less. The incestuous grandchildren are the last hope for his line to "legitimately" be King, as they are accepted as children of King Robert Boratheon.