Why do old gun fight scenes sound so weird and cartoonish?

To answer this, let's take a quick look at Merriam-Webster's definition of "art", specifically the section on synonyms:

ART, SKILL, CUNNING, ARTIFICE, CRAFT mean the faculty of executing well what one has devised.

If we look at the art of foleying—for this is what we're talking about—it's sort of interesting to note that it's always been at least as much a fashionable thing as a technological thing. In the earliest radio and recording days, there were very simple techniques used to foley, though I think nowadays we tend to underestimate how cunning the early artists got.

This Mystery Science Theater 3000 sketch is one of my favorite lampoonings illustrating your very question.

Now, why does anything sound the way it does in a movie? Do they record the actual thing in the actual environment and then play it? Almost never. And when that is done, it generally sounds cheap and lacks any kind of emotional impact which is after all the point of effects and movies generally.

Let's take a really blatant example that will reflect back on the gunshot question. In the '50s, a realistic rocket sounded like this. Fairly close given the limitations of the technology, right? And Rocketship X-M is mostly silent in space, but not completely—because boring!—and then in the late '60s with 2001 and Planet of the Apes, you got a lot of silence in space.

Then, of course, Star Wars comes along and all of a sudden ships are screaming through space right and left, and basically destroys the "silent space" thing for all but the most serious of sci-fi. The audience's expectations were forever changed.

Gun and fighting sounds present a bigger, subtler problem. If you've ever heard a fight, for example, you know it doesn't sound anything like a movie. If you've been in proximity to a gun, you know that not only does sound equipment not capture it, you probably wouldn't want to be in a theater where it was duplicated (as it would hurt your ears terribly).

So, if you're a foley, what do you do? You're not working in a vacuum. You can't just "be realistic", because—to answer your question finally—the audience won't buy it. What you have to do, most of the time, is what the audience expects. At the time, those KAPOING, POW and ZOINK sounds were shorthand for "this is an exciting and dangerous (but also fun) gun battle".

I often wondered where the ricochets were coming from, myself. What had been hit, and where had the bullet been deflected?

But you really answered your own question: Gun battles sounded like that because that's how gun battles sounded. It's why cars of the era were guaranteed to explode into fireballs when shot with bullets—because that's what cars are supposed to do when shot.

The more provocative thing to realize is that movies today are just as artificial in their tropes and effects, and a few years from now audiences will look back on current year movies and be just as amused.

As far as the high-pitched noise goes, you might want to think about what's being shown. The sound you're hearing is (supposed to be) not the sound of the gun itself, but the sound of the bullet ricocheting off a surface. In your For your eyes only clip, the surfaces are entirely concrete and metal, and it's reasonable to expect ricochets. In Casino Royale, the surfaces are predominantly drywall and wood, and bullets will tend to penetrate. Rogem's Finnish film was set in a forest.

For most of us, we aren't in a position to confirm what a ricochet off a hard surface sounds like. However films from the 1950s have similar sounds, and both film-makers and audiences back then had personal experience of hearing that with real bullets. Whilst moviegique's answer is good, do consider that Foley artists generally aim for what things actually sound like, and only deviate where there are artistic reasons to do so.

On a similar theme, in Lord of the Rings Peter Jackson tried giving his actors direction for the scene (in the extras; cut from the theatrical release) where Wormtongue stabs Saruman. Christopher Lee interrupted to say "Do you know what someone getting stabbed in the back sounds like? Because I do." Christopher Lee of course served in Military Intelligence in WWII and was seconded to various Special Forces operations.

This question is good, and the posted answers are good, but looking at this from a far higher level you need to realize something:

A movie is make-believe and sound effects and special effects are never “realistic” as much as they are artistically believable.

So yes, in the past films looked, sounded and were acted in “clunky” ways compared to modern films. But that will always be the case as creative technology improves But still, at the end of the day, if what you are seeing and hearing on the screen is believable and allows you to suspend your disbelief, that is all that is needed to make a film work.

Modern films seem more realistic due to better film technology, sound technology and post-production tools. But at the end of the day all of what you see is a fabrication and a fantasy. And that old clip from For Your Eyes Only (1981) has tons of unrealistic things happening in it that are far more laughable than the sound of gunshots; look at how people die or get knocked over by “the hero” in one punch. Nobody watching a film like that wants realism: They want escapism.

The reality is old films from the early days of cinema can still be compelling even with primitive technology. As long as you can buy into the fantasy, it seems completely realistic to you and not much else can be said. It’s kind of like riding a roller coaster: Is that a realistic way to present railway technology or is it a cool ride where you know you are going to take a trip on a crazy train track?

Old Hollywood sounds were the same as today, which were recorded sounds. However, sound designers usually used rifle or cannon sounds to get a more "punch at the face" and "larger than life" gun sounds because handgun sounds were too weak and they sound like firecrackers. You also need to keep in mind that microphones were not as good as today's microphones. When they got enough recordings, they would process them with their primitive technology.

As for today's Hollywood, those "realistic" gun sounds aren't realistic at all, except for sounds in Heat (1995). The sound designers also try to follow the same concept of "punch at the face" and "larger than life" gun sounds.

Why do they make these handcannon sounds? Well, it's because it would make the gives the audience the feeling that the characters have some sort of power with the gun, and would also make the guns more powerful and deadly.

By the way, I do own a gun sound libraries bought from Sound Ideas, Dynamic Range Sound Effects Library. They have a ton of recorded and processed gun sounds. Just to let you know, there are some decayed handgun sounds which sound like handcannons. They don't make them for fun. They make them with actual purpose. Moreover those processed gun sounds require additional processing if the user wants to synchronize it in their movie project.

In the end, please remember one thing: gun sounds in Hollywood aren't meant to be realistic but rather artistically impactful but believable enough. Exaggeration is the key.

I would wager that spending nine hours studio time recording the same M-16 from different angles and in open and enclosed environments is really costly.

With the exception of hunters, veterans, gangsters, and farmers; I would guess most civilians couldn't tell the difference between an M-80 firecracker and a 12 gauge pushing buckshot.