How did the detectives identify that Russell Nash is from the 18th century just by his signature?

The detectives are using forensic document examination, which identifies consistencies in handwriting to establish documents written by the same person. They determined that all the signatures in the screenshot were by the same person, who currently calls himself Russell Nash. The first signature is from the 18th century, therefore "Russell Nash" was alive in the 18th century.

It is a genuine technique still used by police forces and investigative organisations the world over, and has yielded convictions; perhaps most famously (of late) assisting in proving the guilt of Robert Durst, as featured in the documentary series The Jinx.

The subtleties of one's handwriting, especially how one writes, are largely subconscious and even if someone is attempting to conceal their handwriting, they can often be given away.

This technique is most commonly used in identifying forgeries; spotting inconsistencies as opposed to similarities.

They didn't actually determine that he was from the 18th century solely from his signature. What they did was use comparison of the letters in the signature to show that the wills that had been used to pass his property down through the generations had all been signed by the same person, and since the oldest of those documents was from the 18th century, they concluded that he had also been around since the 18th century.