What's with Interstellar's strange sound mixing?

As @DisgruntledGoat comments, this was all intentional. There's a fantastic Hollywood Reporter article, which includes an interview with Christopher Nolan, which discusses the reasons. To quote the most relevant parts from the article:

Since the movie’s opening on Nov. 5, some viewers have complained about the movie’s sound, claiming some key dialogue is difficult to hear and raising questions about whether it is the fault of the sound mix or the sound systems in some of the theaters where the film is playing. But Nolan said the movie’s sound is exactly as he intended and he praised theaters for presenting it correctly.

Nolan himself states:

“The theaters I have been at have been doing a terrific job in terms of presenting the film in the way I intended,” he continued. “Broadly speaking, there is no question when you mix a film in an unconventional way as this, you’re bound to catch some people off guard, but hopefully people can appreciate the experience for what it’s intended to be.”

He also stated:

“We made carefully considered creative decisions,” he said. “There are particular moments in this film where I decided to use dialogue as a sound effect, so sometimes it’s mixed slightly underneath the other sound effects or in the other sound effects to emphasize how loud the surrounding noise is. It’s not that nobody has ever done these things before, but it's a little unconventional for a Hollywood movie.”

Nolan also described the scene in the hospital, where Michael Caine's character died:

“The creative intent there is to be truthful to the situation — an elderly man dying and saying something somewhat unexpected. We are following the emotional state of Jessica’s character as she starts to understand what he’s been saying. Information is communicated in various different ways over the next few scenes. That’s the way I like to work; I don't like to hang everything on one particular line. I like to follow the experience of the character.”

So the sound decisions were clearly intentional and seem to be to give a more realistic tone to everything that was going on, involving the audience fully in the scene by allowing them to experience what the characters were experiencing - so, for example, in moments of great stress or shock, the actual dialogue becomes less important and instead we hear either the sound effects of their surroundings or the music Nolan chose.

Yes, it was apparently intentional. I don't have the original source but here is a quote from critic Mark Kermode from the November 21st episode of the Kermode & Mayo podcast:

There's been an awful lot of fussing about the dialogue, right down to the point that Chris Nolan's had to be quoted in the press as saying: "You're not meant to be able to hear all of it. There are moments in which the dialogue is meant to be drowned out by the sound effects. That's how I mixed it."

This occurs at 41:15 in the episode.

I'd like to agree with Andrew Martin's answer, and to offer my own interpretation of the quirky sound mix.

The voices are often accompanied by music, for example a flute synthesizer pad held on C5 kind of range, same as a female singing range... the music is often as loud or louder than the voice, to make it more emotive, and consequently it less intelligible, because the high frequencies of any voice clash with C5 chord, the ear can't isolate the voice as well. So you have to listen harder, more intensely.

The voices aren't mixed in a balanced way, as in with a flat sound spectrum. they have been heavily biased towards one frequency or other, which means that the voice's dynamic range is distorted, some dialogues are a bit tinny. You can hear spikes in the voice that emphasize the SSS, TSCH, and TSS sounds, a little bit, and mask some of the lower frequency vowels.

Some of the dialogue is in a fairly low voice, a casual half whisper, rather than a full throated voice, which doesn't help for clarity.

Most cinema's use particular care to isolate the vocal frequency spectrum from the other sounds, FX, Music, to keep it very clear. Interstellar is more reminiscent of unmixed and amateur sound mixing, and also masterfully mixed for a characterful tone and increased tension, like a suspense movie.