How do they mix real characters with animated characters?

Mixing of real characters with animated characters is done in several ways (wiki link)-

• double-printing two negatives onto the same release print.
• More sophisticated techniques used optical printers or aerial image animation cameras, which enabled more exact positioning, and better interaction of actors and animated characters.

Example- In the penguin sequence in Mary Poppins, they filmed the live action part first, having the actors sitting in front of a painted background. Then the penguins were added using cel overlay which was then reshot.

For more modern films like Avatar, advanced techniques in green screen and compositing technology have made this process much easier (in the right hands).

Look at this FX reel (skip to 3:00 mins) to see the use of green screen to combine a live action character with an animated character (in this case a sky dragony thingy).

In the case of a film like Who Framed Roger Rabbit, Bob Hoskins would be playing his scenes to a prop rabbit or a tennis ball on a stick to ensure consistent eye levels. The animators (Richard Williams' team) would then go in and animate over the live action plates, with the final images rendered, shaded and composited in computer.

Here's a great old feature that explains a bit about the process:

The older style of combining animation with live-action is called rotoscoping.

As TylerShads says, the newer method is to dress the actor in a "Capture Suit" which tracks the 3D position (in the room) of various spots on the body.

There is an older method of motion capture, where the actor wears a simple suit of a solid color, and the spots to be tracks are simply dots on the fabric in a contrasting color. This gives the necessary geometric information, but the graphics department must do a lot more work to manually track the points in the camera image.

Yet another method is used in the films This Waking Life and A Scanner Darkly. For these, the live-action was filmed normally and then animation was drawn over the top (using the computer to emulate key-frame animation).

Still other methods were explored for creation of the classic Tron, where they employed a black-screen technique more similar to photographic (darkroom) methods than the others here. Various layers of the digital environment were burned into the same film during multiple passes while dodging the actors. This required the camera to be absolutely still during composite scenes with actors. All-virtual cut-scenes were naturally more dynamic by the use of flying cameras.