What type of law is this?

Duelling
I'm not sure it was ever really legal, and as far as I'm aware never involved a court, and certainly not by the time that Barry Lyndon is set (approx 1750 - 1789), but it was certainly a commonly-recognised way to "get satisfaction" if you felt your honour was impugned.

The duel was based on a code of honour. Duels were fought not so much to kill the opponent as to gain "satisfaction", that is, to restore one's honour by demonstrating a willingness to risk one's life for it

Legislation against dueling goes back to the medieval period. The Fourth Council of the Lateran (1215) outlawed duels, and civil legislation in the Holy Roman Empire against dueling was passed in the wake of the Thirty Years' War(1618 - 1648). From the early 17th century, duels became illegal in the countries where they were practiced. Dueling largely fell out of favor in England by the mid-19th century and in Continental Europe by the turn of the 20th century. Dueling declined in the Eastern United States in the 19th century and by the time the American Civil War broke out, dueling had begun an irreversible decline, even in the South.
Public opinion, not legislation, caused the change.

Barry Lyndon is a movie that took place in 18th century England.

"By the early Victorian era, duelling had already been illegal throughout the British Empire, and most of Europe, for about two centuries. However, in most societies it was accepted, particularly between military officers, and the participants of a fair duel were not usually prosecuted. It should be made clear that these “matters of honour” applied only for men of the nobility and gentry, who were somewhat above the law anyway."

"Whatever the case, a duel generally followed an offence, real or imagined. A gentleman would demand satisfaction and signal this with an insulting gesture, such as throwing a glove before his offender, or proclaiming loudly within earshot some flaw in a man’s character or behaviour. The subject of this was obliged to accept the challenge, or admit wrong and apologise, or be dishonoured. Each party would then name a representative, someone they trusted to act as their second, who would work out the details. The seconds would negotiate a duelling ground, the weapons to be used, and a timing, perhaps ensuring a surgeon was present. Fairness between the combatants was paramount, so the weapons had to be considered equal by both parties. This usually meant the same model of pistol was used by the men, but with sabres there could be variations, as a small man may have been permitted a longer lighter blade, the large man something heavier. Whatever, the seconds had to agree the fight was fair, and often used details like this to persuade both sides that a duel was folly, to shake hands, and part as friends. Sometimes reason was completely lost, the seconds duelling as well, and all four men dead on the field of honour."

"There are records for about one thousand duels fought in Great Britain between 1785 and 1845. Of these two thousand combatants about one hundred and fifty died. However, there were probably at least this many fights done in secret, and no way of knowing how many proved fatal. The last recorded deadly duel fought in England took place at Englefield Green on the 19th of October, 1852, between two Frenchmen over some remark about a former girl friend, but was more likely politically motivated. Emmanuel Barthelemey shot Frederic Cournet, who was carried across the cricket pitch to the Barley Mow pub where he died. Barthelemey and his two seconds were arrested and charged with manslaughter, convicted, and served seven months." - Kate Katterall

I can narrow down the time of Barry Lyndon a bit. At one point in the movie Barry Lyndon is serving in battles involving "Our illustrious ally Frederick". Great Britain was allied with Frederick the Great of Prussia during the Seven Years War of 1754-1763, so part of the movie happens during the 1750s or 1760s.

Thus dueling would have been illegal, but something a gentleman was expected by his social code to do if challenged, during Barry Lyndon's lifetime.