Why the Robots were so unconventional?

Creativity.

Personally speaking, the square droids were a welcome change to the traditional representation of robots. The reaction to the first scene with it was like: "Hmm, that's a strange robot". The second scene was like: "Nice, it can walk". A later scene: "Very nice; he can run, grasp and carry too".

I think producers often have the challenge to make things appealing for the audience. And with this creative alternative of a robot, in a less is more way, they succeeded in creating something square to be a totally convincing personality.

I've seen an interview with director Christopher Nolan quoted in several places, including this one. Here are some of his key quotes:

I wanted to have a machine in the film that was like a piece of gear -- very tough, very resilient -- that had been designed for whatever purpose best suited it.

...

My idea was to remove any trace of anthropomorphism, so it doesn't have a face. It doesn't have arms and legs. It does have a voice and, therefore, a personality.

...

In my brother's draft, he was really into robots and artificial intelligence. What I wound up focusing on was the issue of why you need human beings on this mission. The robots are presented as being physically superior to humans and able to lift heavier things and follow orders perfectly. We kept coming back to the idea of intuition, human adaptability and innovation. That's driven by a survival instinct, which a robot can't have. That makes the robots very important in the story. They take on their own incredible personalities, but they're not human. They keep you thinking about what it means to be human.

So my interpretation is that they were purposefully inhuman, to direct the audience to the transcendent characteristics that humans have that robots can never emulate. The robots provide contrast to one of the main focuses of the movie, which is the power of love and relationships. (I don't think I've ever seen an astronaut in a film sob because his child misunderstood his loving intentions.)

While I'd wholeheartedly agree with NGLN's assessment, there might be some additional things to consider (even if Brett's rather authoritative answer as already touched them to some degree):

First of all, seeing that an obvious connection many people draw in this movie is to Stanley Kubrik's 2001 - A Space Odyssey and the many other slight references to it scattered throughout Interstellar (well, what movie of this kind wasn't influenced by that?), you might really not be too far-fetched in your first feeling that it reminded you of the famous monolith from that movie. It might very well have served as a stylistic influence

mostly it's tried to make robots more humanly looking but they made them squares.

But I'd rather ask you, why should it look more humanly. There isn't so much of an obvious advantage to take out of a more human appearance. In fact from the footage you saw of those strange creations it becomes clear that these robots are highly adaptive and functional despite (or because?) of their rather simplistic form.

In this case I'd yet again make a reference to Wolfgang Schmitt's interesting analysis of the movie (unfortunately not available in English, though). And while he can sometimes shoot a bit over the top with his analyses I agree with him when he emphasizes the rather down-to-earth (no pun intended) pragmatism prevalent in Interstellar. And I think this fits well to this simple but highly functional robot. In fact he even goes as far as to say that the movie with its rather non-esoteric and non-complicated story is pretty much a counter draft to 2001. At no point does the movie try to answer any deeper philosophical questions, let alone ask them at all, it is a straight-to-the-point story about humans and their progress. And for Schmitt the depiction of the robots and the humans' interaction with them is a welcome alternative to the post-humanistic attitude of imbuing souls into each and every thing he sees prevalent in modern cinema.

The robot can have as much of a human voice, it still remains a useful tool, nothing more.

Even if the robot simulates emotions and behaviour (and goes out of his way to parodize 2001 with his humorous comments), those emotions are still freely configurable and primarily there for the humans' comfortability. To quote Cooper himself:

He's a robot, I don't have to ask him for anything.

So basically the robot is not supposed to look like a human, because it simply is none and is not to distract from the real humans of the story. It is a tool made to perfectly adapt to a variety of circumstances while staying as simple in its structure as possible, a very advanced iPhone smart phone if you want.