Why is the punishment for regicide so arbitrary?

You are forgetting the main part. Ned Stark confessed to the crimes he was accused of. Tyrion didn't confess and demanded a trial by combat. So Tyrion's outcome depended on the outcome of combat, whose rules are here.

Of course, the reason he lost his head was becuase of sadistically naive Joffrey. Cersei would never have beheaded Ned. She knew the consequences of his death, i.e. war.

After Sansa pleaded for mercy with Joffrey and Cersei to not kill Ned and just send him to the Night's Watch, IIRC, it was Varys who broke this news to Ned in his cell. Something along the lines of "Sansa pleaded for your life. Everybody was so moved by her emotions. You will have to choose between your honor and your daughters' lives".

Ned had no choice: if he stood for trial as Tyrion did, he would be found guilty. Plenty of evidence in the form of Littlefinger and the bribing of the Gold Cloaks was there. If he stood trial, he would be found guilty and lose his life as well as those of his daughters (certainly Arya, not Sansa).

So Ned, for the 1st time in the series, put aside his honor and chose selfishly for his family. And with the irony that is the Game of Thrones story, he would lose his head for it.

Also, Jaime wasn't a hostage until after Ned was beheaded. The bargaining power was for his daughters.

If everything had gone according to plan, the outcome of both trials would have been the same. In both cases:

• The accused initially denied the charge of treason
• Eventually, a deal was struck in which the accused would "plead guilty"
• As part of the deal, the accused would be allowed to take the black and go off to the wall.

Had everyone involved in both trials been level-headed, both Eddard and Tyrion would have ended up serving life sentences as part of the Night's Watch; given often those pulled into the Watch are described as "rapists, murderers, and worse", it seems like banishment to The Wall is not an uncommon punishment for extremely serious crimes.

The reason the two trials did not play out as described was because of the differing ways that the "deals" went off the rails.

• In Ned's case, Joffrey's last-minute decision to order him beheaded is what cost him his life. All the way up to that point, everyone involved assumed that Ned would be leaving with Yoren and heading North.

• In Tyrion's case, the decision to put Shae on the stand cause Tyrion himself to break the deal Jaime had struck for him and demand a trial by combat. Of course, if Tyrion loses the trial by combat, he will be found guilty and executed the same way Ned was, but if he wins, he will be automatically ruiled "not guilty."

Note that Ned actually had the same right to demand a combat trial as Tyrion, but he was smart (?) enough not to risk losing when there was a guaranteed way to make it out alive. He didn't count on having someone in Joffrey's ear angling to get Ned killed anyway, so in a way, yes he was just unlucky.

Historically, both in fiction and in reality, killing the head of state changed the government. The new government would then get to choose the punishment as they saw fit. If the old king was a tyrant that no one liked, the assassin would possibly receive a parade, statuary etc. Sometimes the assassin intended to become the new head of state and thus above the law. A good example here is Mary, Queen of Scots. Considered the rightful queen by English catholics for complicated reasons, Mary attempted to kill Elizabeth I and claim the throne. Many similar situations existed in Roman days, but it was almost an accepted practice back then.

If the recently-deceased ruler was well liked, or the plot fails, those involved can expect a rather unpleasant time. Mary was beheaded and the rest of the plotters were drawn and quartered for their efforts.

History is written by the victors, so when you "win" a new government you can write your own. Thus, the punishment for regicide is indeed rather arbitrary.