Why do soap operas have the soap opera effect?

You've got it backwards. No effect is being applied to soap operas. It's the tvs that are applying an effect. Let me explain.

To fool the human brain into believing that a series of flashes are a continuous, moving image, the flashes have to come at a minimum of between 50-60 Hz (cycles per second). The chemicals available in the early days of motion picture film simply weren't fast enough to get a good exposure at that speed--and it would have taken a LOT of film. (In 35mm, that's almost 1.5 meters of film going through the camera every second!) It was realized that you could cheat by flashing the same frame multiple times. Standard motion picture film, though only shot at 24 frames per second, sits behind a spinning shutter that looks something like a radiation symbol: As it spins, it flashes each frame three times: 3*24=72 Hz.

When video came along, it used a different method of cheating. Video cameras would record (and tvs would play back) every other line of an image, alternating between even and odd at (in North America) 59.94 Hz. This means that effectively tv was playing back motion at ~60Hz, rather than flashing the same frame 3 times. It's a subtle effect, but it's one that our brains can sort-of detect: The motion in a native video image is "sharper" than film.

Modern HD tvs can display full frames of video at 60 Hz. To show off this feature, some idiot one realized that, by adding a cheap frame interpolation chip, any source--including movies shot at 24 fps--could be nudged up to 60 Hz. (I say cheap because the same frame interpolation was already available as a transition on the first generation of DV camcorders back when I was in college.)

Where soap operas come in is their speed of production. To save time, they've always been shot with studio cameras, on video. Contrast this with bigger budget shows like "Gilligan's Island" or "M*A*S*H", which were shot on 16mm film and then transferred to video. It's entirely cultural. We've grown up with "big budget" material having that slower, more dreamlike motion of low frame rates, while soap operas have a sharper, jarringly more real motion. That's the "soap opera effect."

The soap opera effect is the result of budget limits and production constraints which soap operas are produced under - including the cameras and lenses used to capture soap opera performances. As daylight television does not receive as many viewers as evening programs, less advertising revenue is generated, and hence, less money is spent on the productions. The primary contributing feature which is responsible for the "soap opera look" is the lighting of the scene which needs only to be adequate for coverage, as opposed to maximized for consistent aesthetic appeal. As far as I know, soap operas have never been shot on film - except for maybe rare exterior shots in more popular series such as The Guiding Light (which also was the first show to start shooting in HD in 2008). The idea is to spend as little as possible and like their radio counterparts, the emphasis was on live performances.

Computer generated interpolation is a possibility albeit a time consuming and expensive one and thus unlikely to be used on budget conscious productions which do not feature any CGI.

More likely, shows shot on film and "post-produced" prior to broadcast and theatrical release use a "three-two" cadence of "pulling out" frames from lower rate footage (e.g. 24fps 35mm film) into higher rate NTSC video (e.g. 29.97 frame per second, aka 59.94 fields per second). This method was developed such that film editors could transfer 24fps to NTSC video, cut the movie in the less expensive video format and then make edit decision lists which could be matched back to the original camera negative frames for printing theatrical releases on 35mm film stock. To address any noticeable "effect" of 2:3:2:3 pull out, there are 2:3:3:2 and 3:2:2:3 cadence options. This kind of "pull out" (not interpolating existing frames into new frames, except in the sense that fields of adjacent frames are combined into a technically "new" frame which gets dropped when matched back to film) is much easier to achieve in real-time rather than analyzing frames, creating inter frames, applying motion analysis and rendering new frames based on all that mathematical computation. Such, however, is no longer the usual case in the world of HDTV & UHDTV except where acquisition is done at 24fps and distribution done at a higher frame rate. These techniques, however, are also not used for the production of soap operas.

The plasticity you describe is, as a viewer, simply the result of not being used to higher frame rates and higher resolutions of HD formats which can make this quality more pronounced. Don't take my word for it though, watch this clip at 1080p60fps and then watch it at a lower resolution, such as 480p or lower:

This plasticity is also very noticeable when comparing the experience of watching the intermittent frame rates of 24fps film to the experience of watching continuous streams of interlaced or progressive frames. In short, video engages our persistence of vision differently than discreet frames of film. When watching 24fps film, you are also often sitting in a dark theater, so you're making use of "night time" vision and the experience is inherently surreal, the viewer much more sympathetic to suspension of disbelief.

Think of a flip book animation:

It only take about 18 frames per second to trick the brain into imagining continuous motion. Modern projectors are much more advanced, but in essence, when watching projected film, you are spending about half your time in the dark. Each frame gets projected onto the screen, then a shutter closes over the projector while the reel is advanced and the next frame held in place. The shutter then opens, projecting the images and the cycle repeats. While the shutter is closed, the viewer is in the dark and the flashed image persists momentarily.

This is quite different from how video is streamed. Also, with film, you are seeing a reflected light source. With HDTV, you are looking directly into the light source. But, this streaming along with the higher resolution of pixels available in HDTV and UHDTV, is what gives the images their plastic-y look. It is similar to watching footage shot with a high shutter speed - such that the motion blur of each frame is minimized and each discreet frame is sharp. You will notice in films with less expensive computer generated visual effects that motion blurring is heavily relied upon to sell the look.