Is there a criterion to differentiate "thriller" from "horror"?

Your observation is correct: "thriller" and "horror" are often thinly separated in the already vague world of film taxonomy. There isn't really One True criterion for distinguishing the two.

That said, there are genres, and the thriller and horror genres can be seen, Venn diagram-style, as not completely overlapping. Check out the Wikipedia entry for thriller https://en.wikipedia.org/wiki/Thriller_(genre):

Thrillers are characterized and defined by the moods they elicit, giving viewers heightened feelings of suspense, excitement, surprise, anticipation and anxiety. Successful examples of thrillers are the films of Alfred Hitchcock.

Versus horror: https://en.wikipedia.org/wiki/Horror_film

Horror is a film genre seeking to elicit a negative emotional reaction from viewers by playing on the audience's primal fears...The macabre and the supernatural are frequent themes, and may overlap with the fantasy, supernatural fiction and thriller genres

Note that the words associated with "thriller" are uniformly positive. A thriller is "exciting", "suspenseful", etc. It tries to "surprise". The category of horror movies that fail to invoke "feelings of suspense, excitement, surprise, anticipation and anxiety" is "failures". Even the worst horror movies will try at least to shock the audience, and what is "shock" but a subcategory of "surprise"?

A successful horror film must, therefore, give us some aspects of the thriller. (And Michael Jackson was not completely off-base. :-))

Part of what's going on was first described by Mrs. Radcliffe:

Terror and Horror are so far opposite that the first expands the soul, and awakens the faculties to a high degree of life; the other contracts, freezes and nearly annihilates them.

The latter is the "negative reaction" described in Wiki, and would be the key element in distinguishing a horror film, if you wanted to really strictly classify things.

As Dean Koontz points out when he's denying being a horror writer, literary horror has a tradition rooted in nihilism, and while it's not nearly so strict for movies, it is still there. Koontz writes thrillers with horror effects. Horror requires a kind of futility in action—that contraction of the soul that says "this is beyond your capability to fix, or even survive"—seen in Poe, Lovecraft, Dunsany, etc.

And we can see this kind of concept works with movies as well. The "Resident Evil"/"Underworld"/survival-horror genre are action flicks with horror effects. "Love at First Bite" and "The Lost Boys" are comedies with horror effects. Today, horror effects are broadly used in romances ("Twilight", anyone?), superhero movies ("I, Frankenstein"), and kidfilcks ("Hotel Transylvania").

So, is there a rule of thumb that can be applied? Perhaps this: A thriller involves a (as Hitch called it) "MacGuffin": The thing everyone wants that motivates characters' actions. There is no "MacGuffin" in a horror film—even when the characters sometimes think there is, which is a common horror device—beyond the existence (biological or spiritual) of the characters.

"Psycho" isn't a thriller about embezzled money; it's a horror about a knife-wielding maniac. "Hellraiser" isn't about LeMarchand's box, but about the souls of those who touch it. "Friday the 13th" isn't about campers having sex and doing drugs, but about their ultimate demise.

I hope this has been helpful.

Thriller:

a novel, movie, etc., that is very exciting : a story full of exciting action, mystery, adventure, or suspense

Horror:

noun: something that causes feelings of fear, dread, and shock : something that is shocking and horrible
adjective: calculated to inspire feelings of dread or horror : bloodcurdling < a horror movie >
noun as used in definition below: a very strong feeling of fear, dread, and shock

Bloodcurdling:

causing great horror or fear

Thus, a horror film that exciting is also a thriller (and a horror film that's not a thriller may just be badly made or go too far causing fear/shock/dread past the point of exciting its audience), but not every thriller is a horror. For those that fall into both categories, it is up to the speaker to determine which descriptor seems more primary, according to what was more important in that person's experience of the film (or for the producer, what s/he intends to be primary in the audience's experience). When both adjectives seem equally primary, the one that is narrower or more specifically descriptive / covering a smaller set ("horror") is used.

I can answer this question for you.

The difference between the horror genre and the thriller genre is, as has been mentioned here already, pretty subtle, but different enough to matter.

The job of a horror film is to take the viewer through the experience of trauma.

The job of a thriller film is to take the viewer through the experience of tension and release.

These subtle but important differences are why the two genres are hard to define. Depending on how these two experiences are explored, the two can seem strikingly similar in a film narrative. That is also why there are so many horror thrillers out there.

I'm going to emend my original answer: Horror films try to invoke a fear or awe of the otherworldly or transcendental; whereas thriller films try to scare us with more mundane things.

For instance, "Repulsion" by Polanski is a horror film with no overt supernatural elements to it. But what scares us in the film is not the knowledge of a young woman having psychotic episodes, but instead the otherworldly visions in the episodes themselves, the hands coming out of walls, the person in the mirror who isn't there.

"Texas Chainsaw Massacre" is a type of horror film with no overt supernatural elements to it, but I believe it's horror because of the completely over-the-top, out-of-the-ordinary events and artifacts, such as the masks of dead flesh, the cannibalism, the bone structures; etc.

[Original answer: I would guess "horror" films would tend to contain one or more instances of supernatural elements, whereas "thriller" films would contain more or less real-world type elements exclusively.]

A thriller is a crime story from the point-of-view of the victim or potential victim. A horror film is a thriller, but the antagonist(s) are either supernatural or extra-terrestrial or in some way diabolically monstrous beyond what you expect from a regular thriller.