Who are the Guys in Tall Black Hats in movie 'The Greatest Game Ever Played'

I believe they are flashbacks to the opening scene, in which young Harry is told that golf is a gentleman's game and therefore he shouldn't wish to play. He has now become the best golfer in the world, but he always remembers the day they told him he wasn't good enough.

While Ouimet is the main protagonist, Vardon is played up quite a bit as a sympathetic character because they both had to work their way up through external discouragement.

The men in the black hats are judges. There's one judge left himself. He makes them disappear because the only judge that matters is ones self. He does not judge himself, he does his best. So he makes the last judge disappear.