Why does the boom operator have to hold the boom mic by hand?

According to this patent application by Oscar winning microphone specialist Les Drever (whose award was for "technical achievement in the field of microphone windscreen and isolation mount design"), the thinking behind the hand-held boom (AKA 'the fishpole') is largely one of convenience and cost...

A hand-held or fishpole boom is physically held and guided by a single person (the boom operator). A microphone is mounted at one end of the boom in a fixed angular position, with the other end of the boom being manipulated by the boom operator to position and orient the microphone. The hand-held boom has the advantages of being relatively simple to operate and fast to manipulate in the hands of a skilled operator. It will fit into tight spaces, and is essentially not limited by the required distance of movement, since the operator may move as well as moving the boom. It is rugged and highly portable, as well as being of relatively inexpensive construction, and is in world-wide use.

He also offers some disadvantages...

The primary weakness of the hand-held boom is that the microphone is fixed at a predetermined angle, which can only be changed by interrupting the filming or taping sequence. Only crude sideways cueing is possible by twisting the boom, and alternating forward and rearward aiming is only possible by bringing the boom around in a 90 degree arc, which is not very practical. With complex work, two or more booms must be used, which means that two or more operators will be necessary as well. Even with a single boom and microphone, some movements must be coordinated, requiring for example a second operator to look after the trailing microphone cables. In addition, the hand-held boom becomes tedious to support by hand over a period of time, particularly when the boom is being used with a heavy directional microphone.

OK, I'm going to posit this as an answer… though there is some cynicism…

Edit: Let me add some late specifics...

In the first picture, the girl a) has got a comfortable leverage grip on the boom b) she's used to maintaining that position, it's her job, she does it all the time & c) that "big mic" isn't big at all, it's a small, light mic inside a wind-shield... known as a dead cat.

The boom itself is made of carbon fibre & weighs ounces.

In the 2nd picture - that looks more like the guy is at rest between shots. As they get ready to roll, he will extend the boom to dimensions more akin to the first picture & adopt a similar posture.

The third picture looks a complete nuisance - very restricting.

OK, back to the cynical original...

When filming, picture is everything, sound is the 'red-haired step-child' [poor PC-awareness aside]

The Director and DOP [Director of Photography] will spend as much time as time/budget allows getting the shot right, as regards lighting, angle, lens… heck even getting the actors to behave…

Then they hand over to the 1st AD and suddenly it's all "Lights, Camera, Action" [not real, see Who will say "start camera action" when the director casts himself in the main role? ] and sound has seconds to try figure out how to get every nuance of the performance without dropping the mic into frame.

OK, honestly it's not quite as cruel as that; but sound is almost the last consideration when shooting. They have to quickly adapt to a changing action and frame and there simply isn't time to have a comfortable setup to capture the sound.

On a tracking or steady-cam shot there could be as many as 8 or 10 people all huddled in behind the camera, shuffling along. The sound girl/guy has to be able to move with this crowd and still reach the action.

The best that sound can do to ameliorate the aching of their arms is to shout "wait for the plane" when Heathrow is throwing its finest into the air and hope to hear "Hold the roll" from the 1st AD ;)

On the upside, those modern booms are made of carbon fibre & are nowhere near as heavy as they look.

Firstly the assumption about it being tiring actually isn't that accurate. Boom operators with some years of experience can quite happily swing a pole for 5 minutes at a time with a break in between (to reset the shot) all day - lots of it is technique and using the correct muscles and posture in your arms and core and learning how to move between these without actually moving the mic - basically it's not as tiring as it looks if you're used to doing it correctly. (I had a 36 minute take a few weeks ago on a documentary and I could have kept going - and I'm a young member of the profession.)

Advantages - you can match picture perspective, mainly, is the biggest one, the boom op can make things sound as close or as far away as the frame will allow, allowing for a ridiculous amount of customisability over how the dialogue sounds with relation to what the viewer can see... if you are using radios (the correct term is lavalier - often abbreviated to lav) you have no choice, you have a very intimate perspective, very chesty and overly close and in your face which is not at all desirable on the majority of shots (for fiction anyway - for news or documentary or other non story telling stuff this can be exactly what you want) - and thus understandably the majority of sound people in filmmaking, both post and production sound, will prefer a boom if it's operated correctly - it just "sounds" like what your eyes are telling you. If you've ever seen a silly great wide shot with people in the distance and heard them like they're whispering over your shoulder you'll get what I mean.

Also the boom mic is always the highest quality mic in the shot - small diaphragm hypercardioids or shotguns are what we usually use, the most expensive ones are 2000£ each and they have a sound to match their price tag, they have a larger diaphragm than lavs (most expensive will run you around £300 per actor) and capture a more open sound with more detail than something smaller than a pea, especially when operated well and double especially when you're mounting lavs under clothes and cutting off some of the high end (lav mounting by the way, is also the boom operator's responsibility on the set and is an entirely different mountain of skills and practice to swinging a pole - if you sat there wondering why most self shoot documentaries with no sound man on sound like dogshit with no vocal definition and drenched in clothing noise)

In summary - assuming the shot is tight enough to get a boom close enough to get a good signal to noise ratio (critical distance is the proper term) and it's operated by someone who can catch all the lines on one mic and have it in the correct positions - the boom will ALWAYS be the mic of choice - lavs are problem solving devices, used when the shot is too wide or for some other reason the scene is impossible to cover on a boom - even then the seasoned sound team will likely try and cover everything on boom in different shots and angles and it will all be cut together by the dialogue editor, radios are a last resort whenever I'm recording at least.

Finally to offer criticisms to your other suggestions:

2. Hanging a mic is what we call a plant mic (along side planting it anywhere else, on a stand or stuck to a wall or on a friction arm or whatever) and we do use them, but they obviously again can't match perspective and can't move with the actors, generally they're used again where you can't get a boom - or if you have a person sitting down and being interviewed, often you will find a mic hung or on a stand, no good wasting a talented technicians boom swinging efforts on sit down interviews.

3. Again you can't really move with it and chase actors across a floor, as I said for people that know what they're doing it's really not that much effort and the lack of mobility you would get with that thing would drive me f*cking mad - quite blatantly a product designed by someone who's never had to operate a boom professionally in their life - since if you could afford to not move, why not put it on a stand or plant or hang it, as earlier.

4. Massively unnecessary, expensive, taking up too much room, you wouldn't fit a boom track and a dolly track in the majority of spaces you need to shoot, you don't need the same level of smoothness of motion with a mic as you do with a dolly / camera so again it's just easier to hold it, tracks are a pain in the arse to set up and require balancing by a dolly grip and he's also needed to operate it safely, and the whole point of it is that cinema cameras weigh a fucking ton (like 10kg for an alexa? I don't have the numbers) - camera ops can't hold that for 20 minutes at a time, and even less so move it slowly and smoothly. not a problem for a boom op - the mics weigh a fraction of that and are very handle-able.

Why has the film industry chosen to stick with a physically tiresome, incredibly old fashioned method, when better options appear to be readily available?

They only appear better to the untrained eye (which I say with the greatest sense of respect and humility - people often say to me "god that looks tiring" "don't your arms hurt" etc - and no, I'm fine actually).

I am the woman from the picture.

1. Record as many separate tracks as you wish with one live mixed master. Give your editors as much as possible to work with.
2. Lavs pick up clothing noise and are frequently the wrong prospective. (Yes, they are wearing them, and we are recording it, but it is a sword fighting scene, they are moving like crazy so the final recording isn't great.)
3. I am on the ladder because this was a wide angle tracking shot. Left or right of camera meant that my fixed length pole couldn't keep up with the action, I would have to run with the shot while trying to stay out of frame and I couldn't get high enough to keep the pole out of the corner of the shot. Below wasn't an option, there is a dollied camera down there on a wide enough to see me on the ground and keep me too far from the actors. So, there I am on the ladder being able to pivot to follow them, and I have about 3ft of depth I can play with by moving the position of my arms (which the 'lever mechanism' discussed above doesn't allow me, even if it could be operated safely at that height.)
4. Someone asked: why not have bigger heavier equipment? We're on a dirt cliff-side in Malibu in someone's backyard, through their human-sized gate. Get it there, how? Budget for it...look at the dolly setup. And dirt, so much joint destroying dirt.
5. Boom poles are a teeter totter. Your front hand is the pivot point, your back arm is pulling down to keep the other end up. Read more about lever physics, if you are interested. Between shots, head or shoulders to rest, if it's a longer break, shorten it, it collapses and extends in about 20 seconds. I also flipped the ladder around, so each side got a chance to be pivot/pulldown. (https://ki42.github.io/domgee/)

The alternative to hand held booming was traditionally the fisher boom http://www.jlfisher.com/booms/index.asp These are expensive, require significant training and practice, and are restricted to studios. Outdoor locations would allow sand and dirt etc to get in the mechanisms, causing problems. Repairing the mechanics of these tools is also beyond the scope of average on-set personnel. Hand held booming has all the advantages previously described. Fisher booms are still used for multi camera scripted drama and comedy, where the extra long reach and practiced / coordinated movements are a time saver.