It's widely known as Breaking the Fourth Wall.

Breaking the fourth wall is when a character acknowledges their fictionality, by either indirectly or directly addressing the audience. Alternatively, they may interact with their creator (the author of the book, the director of the movie, the artist of the comic book, etc.).

Wikipedia elaborates about the fourth wall:

The fourth wall is the imaginary "wall" at the front of the stage in a traditional three-walled box set in a proscenium theatre, through which the audience sees the action in the world of the play. The concept is usually attributed to the philosopher, critic and dramatist Denis Diderot. The term itself was used by Molière. The fourth wall illusion is often associated with naturalist theatre of the mid 19th-century, and especially with the innovations of the French director André Antoine.

The restrictions of the fourth wall were challenged in 20th-century theatre. Speaking directly to, otherwise acknowledging or doing something to the audience through this imaginary wall – or, in film, television, and video games, through a camera – is known as "breaking the fourth wall".

_{In addition to Walt's great answer, I'd like to add...}

Losing the "suspension of disbelief"

When something happens that "breaks the illusion of fantasy" for the audience, you could say the audience is no longer willing/able to suspend their disbelief.

Definition:

Suspension of disbelief or willing suspension of disbelief is a term coined in 1817 by the poet and aesthetic philosopher Samuel Taylor Coleridge, who suggested that if a writer could infuse a "human interest and a semblance of truth" into a fantastic tale, the reader would suspend judgement concerning the implausibility of the narrative. Suspension of disbelief often applies to fictional works of the action, comedy, fantasy, and horror genres. Cognitive estrangement in fiction involves using a person's ignorance or lack of knowledge to promote suspension of disbelief.

_{- source}

Several things can cause audiences to no longer suspend their disbelief, including:

• Meta-jokes

• Self-referential jokes

• Jokes about jokes ("meta-humor")

• Meta-reference, a metafiction technique, is a situation in a work of fiction whereby characters display an awareness that they are in such a work, such as a film, television show or book, and possibly that they are being observed by an audience. Sometimes it may even just be a form of editing or film-making technique that comments on the programme/film/book itself.

• Breaking the fourth wall

There is certainly some overlap between some of these points. It should also be noted that, while breaking the fourth wall and the other points listed above are predominately intentional, breaking the suspension of disbelief can often be unintentional as well. If the filmmakers fail to infuse enough semblance of truth into their fantastic tale, the audience is unable to suspend their judgement concerning the implausibility of the narrative. For example, some viewers couldn't manage to suspend their disbelief at seeing Indiana Jones survive a gigantic explosion by hiding inside a lead-lined refrigerator, or Clark Kent keeping his identity as Superman a secret without using a mask, and so on. Other implausible occurrences, plot holes, inconsistencies, or when low production values or budgetary restrictions become obvious etc. can similarly make it difficult to suspend disbelief.

Verfremdungseffekt. This is often translated as "alienation effect", "distancing effect" and increasingly as "estrangement effect", sometimes reduced to simply alienation, distancing etc. and sometimes expanded again to "Brechtian alienation" or "Brechtian estrangement" because Brecht placed such importance on it, and indeed coined the term verfremdungseffekt.

Because of the Brechtian influence, sometimes the German term verfremdungseffekt is used untranslated in English.

In any case it is a matter of deliberately interfering with the classic Coleridgian suspension of disbelief. Most common would be various techniques that "break the fourth wall" which is to say refuse to pretend that there is a wall seen by the characters rather than the audience (used by extension for media other than theatre, such as direct-to-camera monologues, characters addressing fact that the reader will have to turn through pages to see what happens next, etc.) though it would also cover any other deliberate breaking of the constants that aid suspension of disbelief and identification with the characters as something other than fictional, such as changing the mode of a work; a realist style film suddenly becoming a musical, switching between live-action and animation, discussing the marketability of a plot as it progresses, allowing the street outside the back of a theatre to be seen (the second wall!) and commenting on passers-by, addressing the writing style used in a narrative, and so on. It would also cover meta-references that would be so blatant to the audience as to highlight the fictional nature, while remaining "unnoticed" by the "characters" (once one begins to deal in verfremdungseffekt the idea of what characters notice, or that characters even can notice something becomes unstable, and scare-quotes come naturally to the fingers when writing about it).

While this is most commonly referred to as "breaking the fourth wall", referring to the invisible barrier between the audience and what they're watching, I've also seen this referred to as "winking at the viewer".
This comes from the fact that the "breaking of the fourth wall" often takes place via one of the characters winking at the camera. Because this is so common, it has become a name for the general practise which it is an example of. For example:

It’s all one big wink at the audience ... and then Schwarzenegger actually winks at the audience.

Mike Myers is one of the great fourth-wall breakers of his generation, and he peppers one project after another with assorted nods and winks to the audience.

Both of these quotes are from "25 Classic Moments When Movies Broke the Fourth Wall".

On stage this is called "aside". Literally, the character turns aside and speaks directly to the audience. Hamlet for example is filled with them - just search for the word "aside".

The purpose of aside is usually either to clarify what is happening or tell the audience what a character is thinking; or simply to build tension. Shakespeare's audiences were encouraged to cheer and shout to build tension or heap malice or ridicule onto characters through use of asides. Because aside is for the benefit of the audience, other characters in the scene usually seem not to notice or hear asides, though in literal terms they should.

This production of Hamlet uses aside well, starting at about 2:05. Polonius speaks directly to the audience, plainly and in the presence of Hamlet, and even discloses to the audience what he plans to do with Hamlet, but Hamlet does not hear what is spoken aside.

In cinema, looking directly at the camera, talking to the camera, bumping into the camera, etc. is also an aside, since the camera essentially is the audience, and if other characters are present in the scene, they tend not to notice it. Ferris Bueller's Day Off (1986) makes liberal use of aside, sometimes as a substitute for narration; even the villain gets an aside at the end.

Aside is distinct from monologue, because in monologue the character is literally talking to himself, or to animals, or to God, and is not aware of the audience listening.

I argue that aside is not a full "breaking of the fourth wall" as described in other answers. Use of aside generally does not alter the course of the story, it is purely a way to increase the degree of audience engagement. My favorite outrageous violation of the fourth wall, which includes a brief aside to the camera, is this scene in Spaceballs (1987) where the villains search through the video tape of Spaceballs: The Movie to discover where the heroes have gone. This clearly violates the fiction because not only are the characters aware of the fiction, they exploit it to continue the story!