Why are films where the bad guy "wins" at the end so rare?

TLDR:

There are plenty of films where the "bad guy" wins.

Ultimately, the reason why the bad guy wins can come down to a number of reasons, including to be more realistic, to set up a later "good" ending in another film or because it's unexpected, to name a few reasons.

Long Answer (note: there will be spoilers for some films below):

There are really a few different things to look at here. For example:

1. Films where a villain "wins" by physically or literally defeating a good guy.
2. Films where a villain "wins" by psychologically defeating a good guy
3. What exactly is a good guy versus a bad guy.
4. What exactly does winning mean?

Now, the first film that comes to my mind when I think of villains winning is Seven. In it, two police officers hunt down a criminal who is murdering people based on the seven deadly sins. The film ends with the "bad guy" manipulated one of the good guys into killed him, thus fulfilling the "final" deadly sin.

Is this a win? I would consider it so. The good guy's morality was damaged and the bad guy got what he wanted - even if it meant his death. Therefore, I'd consider this under Option 2 above - psychologically defeating the good guy.

Another famous example is No Country for Old Men, where the "good guy" comes across a lot of money at the start of the film, and spends the film avoiding a hitman who is after him for it. He doesn't succeed. He is killed towards the end of the film. We see the hitman get involved in a car crash. He could have died. Instead, he recovers and walks away.

So the good guy is dead, the bad guy survives and walks away. A very clear example of Option 1.

Similarly, The Usual Suspects would fall down this road. But what about The Empire Strikes Back? Or Harry Potter and the Half Blood Prince?. Or The Hunger Games: Catching Fire. The "bad guys" certainly won at the end of those films - but do those count, since they were part of an overall series where the good guys won?

When looking at Option 3, I mention the difficulty in defining good and bad. In years gone by, black and white hat symbolism could make this a simple task. But in modern media, it's increasingly difficult. Take a show like Game of Thrones - how does one determine who is "good" and who is "bad"? Everyone will have their own subjective opinion.

And what about horror films? Quite a few of the Saw films end with the bad guy winning. Many other horror films do. Do we count these?

Similarly, what about Option 4? What is winning. Does Amy win at the end of Gone Girl, a film about a woman who intentionally disappears to heap suspicion on her husband, but returns when it backfires (having murdered another person) and forced him into a situation where he must stay with her. Did they both lose? What about Nightcrawler, where a stringer who films live footage of crimes sets up a scene to kill his partner, so he can film it. He was the protagonist of the film, but that didn't make the creepy ending to this film any more comfortable - but it wasn't supposed to be comfortable.

There is a lot of subjectivity around all of this.

Therefore, I'd propose to set aside Points 3 and 4 for now. If we just focus on obvious examples of victory, through defeating the good guy or psychologically breaking him, there are a number of fascinating reasons why a film might let a bad guy win:

Unexpected

It is unexpected for bad guys to win. We got to the films, and watch TV, to be entertained. Often this involves rooting for the good guys, as many of us identify as good ourselves. Of course, these guys might be rogues (like Captain Jack Sparrow, Han Solo), but their characters are written in such a way as to be interesting, relateable and fun.

When the tables are turned and the good guy loses, it suddenly becomes unexpected and adds shock value.

Cliffhangers

Take a show like Mortal Kombat: Conquest. Its first series ended with almost all the good guys dying. The intention was a second series would have resolved this (supposedly by making the first series ending a dream). However, the show was cancelled. So for all of time, there is a depressing ending!

There are slightly different examples like I mentioned above, where films that are part of an overall story arc can have cliffhanger endings where the "bad guys" win - although this is often to set the good guys up for an even bigger win later on.

Realism

Films like No Country for Old Men displayed "bad guys" winning because it was a genuinely realistic finale that was intended to both shock and horrify. And it worked, brilliantly.

Historically accurate

Imagine an infamous criminal from modern or historical times who got rich, lived happy and survived until old age. He has technically "won" (and quite possibly defeated many good guys along the way). However, as this is a film that's historically accurate this "victory" will be shown, even if people watching the film find it distasteful.

So how do audiences react?

This is very difficulty to objectively judge. Are we basing this on critic reviews? On box office revenue? For films like The Usual Suspects and Seven, they reacted brilliantly. The endings are adored. For others, they are not.

Conclusion:

Ultimately, there are plenty of films where the bad guys win.

However, the biggest problem I believe for studios in deciding whether the "bad guy" should win, is understanding why he is wanted to win. If it is just "for fun", they're much less likely to do it. If it's for a twist ending, for shock value, to reflect the brutality of a situation, or to set up an ending for a later, more positive film - then it's much more likely to be acceptable.

Film studios do care about audience opinion. If they feel the audience will appreciate the ending or if it fits with the theme of the story, then they'll proceed with it.

You asked a number of questions here; as often happens, not all of them got answered. Try to ask only one question per question.

Are there actually rules, written or unwritten, preventing a script with victorious villains ever being made into a movie?

Yes. Do a web search for the "Motion Picture Production Code", also known as the "Hays Code". If you've ever wondered why married people always sleep in twin beds and why the good guys always win in old movies, well, now you know.

Has there ever been a movie where the bad guy wins at the film's conclusion?

Sure. The entire film noir genre comes to mind. In noir films there often are no "good guys" -- there's just a bunch of bad guys. And even when the good guys "win", it's often a Pyrrhic victory. "Chinatown" comes immediately to mind as a film in which the villain thoroughly triumphs.

The horror genre also comes to mind; though often our hero wins out in the end in a horror movie, I can think of numerous examples where the monster wins. "Little Shop of Horrors". "Night of the Living Dead". And so on.

For a writer, it's important to get your audience to care about your protagonist (whether they are good, bad, ugly, or neutral). If they don't care about and aren't interested in the protagonist, and their plight, then they're not going to care about your story.

If you're thinking, well, I'll get the audience to care about the bad guy (etc.), then that other thing is your real protagonist. -- Whatever it is, your protagonist(s) are the "bread winner(s)" for your story.

Killing the protagonist, or having them fail their quest -- especially right at the climax of your story -- basically means you've spent all this time getting an audience hooked into your story, and then you're destroying the part that they care about the most.

It can be a good way to lose viewers and ratings. -- A really good example of this is the Game of Thrones / Ice and Fire series. For a couple of reasons:

1. You can examine it, from the audiences' perspective yourself, every time you start getting to the point where you're feeling empathy for a character, and the ball really gets rolling, and then the character is just plain fucking dead. No recourse. That's life. -- It feels terrible. But at least it has a LOT of protagonists (and occasionally converts antagonists into protagonists), so it helps with the pain. Imagine the series if ALL of the POV characters had died in book 1. It would be a death sentence for the whole series.

2. You can see fan reactions even in the current setup of the series (i.e. there is a lot of "my favorite character 'xyz' died, this show is now dead to me."

Despite those two things, the series is doing very well financially (IIUC), so killing your protagonists doesn't always have to be a deal breaker.

While that's true, it's far easier to pull off a cliff hanger than it is to kill your protagonists. Take the movie The Empire Strikes Back, for example -- the antagonists don't really "win" -- but instead, it ends with a cliff hanger. -- Where you take characters that the audience has built a lot of empathy with, and you don't quite kill them -- you don't quite make them lose -- you just put them on the run, in an impossible situation, with all the walls closing in around them (figuratively) -- and then you make the fans wait -- "tune in next time to see how our heroes get out of this one!" -- this helps to build anticipation. -- It works really well for multi-part stories (and historically, it's worked well with TV serieses that are trying to get renewed for another season...).

Useful Reference: Your Hero: Top Ten Rules (Expanded)

The first, most prominent film that springs to my mind is Star Wars : The Empire Strikes Back.

** spoiler alert **

By the end, a Rebel base has been destroyed.. our hero Luke has just had his hand cut off, Han Solo is in frozen in carbonite. Darth Vader and the Emprire definitely have the upper hand.

Of course this has to be viewed in the context of the trilogy - but taken at face value, the Bad Guys™ win this movie.

As for "how this was perceived by the audience" - many people cite Empire as their favourite of the entire series, and when asked why, they often give variations on the idea that it was the 'darkest', 'broodiest', etc.. Usually the same people who seem to hate Ewoks.

The film Swordfish (2001) has part of its plot around this particular subject. The good guys always winning, the bad guy losing; and the whole idea is for the audience to feel happy.

It's also an example of where arguably the bad guy does win. He does get to head off into the sunset having done what he intended. Okay, he paints himself as a good guy, but given what he does I don't think we can classify him as such.

First one that comes to mind is The Usual Suspects

Without giving it away, the plot takes you in one direction, before an abrupt (and clever) plot twist, and the result is that the main villain (Keyser Soze) wins.

We don't tend to get stories with the villan as winner, we tend to follow the heroic character and so invest feelings in them.

There is a tradition in storytelling that dictate that the bad guy "wins" in the end. This is mostly prevalent in horror/ghost stories and can be found in almost all human cultures.

In most of these types of stories, the story structure is almost always a horror reveal where the story ends with the protagonist realising or confronting the real ghost/horror and dies. This is the typical structure of Asian horror stories (not movies, stories). It is also found in Western fairy tales but is not as common. It is common in Western urban legends though.

One of the most obvious examples of this in movies is The Wicker Man. But it's not quite the perfect example because, like the protagonist, we discover the true horror a bit too late.

A better example is The Skeleton Key. We find out who the villains are quite late in the movie but long before the ending. More to the point, the movie makes us suspect the villain even longer before that. But protagonist lose anyway and the villains get away scot-free (I won't spoil HOW the bad guys win in case you haven't watched it).

Twilight Zone episodes are also structured like this with the protagonist typically meeting their doom at the hands of a horror at the end of an episode.

The first movie that sprang to my mind was Shane, released in 1953. The bad guys don't really win, since they're all dead, but the good guy is implied to be dying as he heads into the sunset at the end. So, per your definition, the bad guys have won.

The movie was received well enough in general, but I can't find any references right off for how the ending was received. I personally know a number of people who were disappointed by the ending, but not enough to be useful as a representative population sample.

The rarity of "Bad Guys" winning: This is my opinion.

Throughout the ages, we humans have always taught ourselves stories of Good triumphing over Evil because that generates a feel-good factor in ourselves. And who doesn't want to feel good and/or be satisfied with the ending?

This is probably the reason why the "Good Guys" win and "Bad Guys" don't, usually.

An example of the "Bad Guy" winning:

An excellent cyberpunk film called Upgrade (2018) (directed by Leigh Whannel and produced by Jason Blum, if that helps) introduces

the real "Bad Guy" - STEM, the chip who we'd think was good all along, towards the very end. The psychological strain breaks Grey's(the protagonist) mind and his consciousness believes the idyllic dream it finds itself in to be real, while STEM reveals this was his (its?) intention all along, as now it has complete control over Grey's body and mind.

This movie has 7.5/10 on IMDB, 88% on Rotten Tomatoes, 67% on Metacritic and apparently 93% Google Users like the movie. This shows that it was well received by the audience, probably due the unexpected plot twist.

Film's based on historically accurate events seldom depict the malevolent power winning. And the "Bad Guy" winning; or at least not losing has been illustrated in many movies. Often direct victory is now avoided to perpetuate sequels. Films where the antagonist won (or at least did not lose)

1. Empire Strikes Back (Empire Defeats rebel forces on Hoth, Vader Defeats luke)
2. The One who Flew over the Cuckoo's Next
3. The Dark Knight (Joker succeeds in tainting the Dark Knights reputation by threatening to tarnish Harvey's)
4. Silence of the Lambs / Hannibal (Lector escapes both situations)
5. Memento
6. No Country for Old Men
7. Wicker Man (Burnt to death)
8. Halloween (Michael escapes)
9. Halloween III: Season of the Witch (Cochran thou dead, his apparent scheme broadcast continues)
10. Watchmen (Veidt's plan succeeds, albeit it's never elaborated if Rorschach's journal was distributed to exploit or expose him)
11. Star Wars: Episode III (Jedi nearly exterminated, Republic reformated into Empire)
12. Phone Booth (Shooter escapes)
13. 1984
14. Chinatown
15. Se7en
16. The Omen
17. Brazil
18. Avengers: Infinity War
19. The Green Mile
20. Nightmare on Elm Street
21. Devil's Advocate
22. Rogue One; Star Wars
23. Clockwork Orange
24. Night of the Living Dead (in fairness this film has no clear "Antagonist", the fear, paranoia and racism won the day)
25. Invasion of the Body Snatcher
26. Phantasm
27. House of a 1000 Corpses
28. I know what you did Last Summer
29. Terminator Franchise (in a sense, despite what attempts are taken, Judgement Day is never stopped, only post poned)