What's the difference between "men of violence" and "violent men"?

In the continuation of this scene, Dr Jeremiah explains that difference:

Dr Jeremiah: No assumption. Not at all. you misunderstand me, I said you are men of violence I'm not accusing you of being violent man It's quite different

Please, please, edify us, Doc.

You both served overseas

Not much of a stretch, Doc. for all you know, we were both paper-pushers over there

No, you were not. Since the schoolyard, I would bet neither of you has ever walked away from a physical conflict. Not to suggest you enjoy it, but because retreat wasn't something you considered an option

Men of violence are people who occasionally resort to violence when it is required - people like soldiers, police officers, bodyguards, or simply people willing to stand their ground. They all can resort to using force (i.e. in their line of work) but that violence is calculated as a necessity, not as an impulse.

In contrast, violent men act on an impulse and resort to violence even when it is not called for and often do this for the pleasure it gives them.

A man of violence will beat you up to restrain you, a violent man will beat you up because you looked at him funny.