I thought you cannot use someone else's character in your own work without the author's permission and (usually) paying for it.

Considering this how did the movie go about using them and not ending up billions of dollars in debt?

They asked permission!

Entertainment Weekly interviewed the director, Chris McKay who explained

EW: Was licensing a nightmare?

McKay: "You need an army of lawyers and producers who are willing to run around and do all the hard work of finding out all the rights-holders. Fortunately, Warner Bros. made a lot of those movies, but for others, we had go to out and get them. I wanted characters from all over the world. I wanted Daleks. I wanted stuff that has a history and is time-tested, like the Wicked Witch. And I hope people love Gremlins as much as I do. And Sauron. But even for the actual Batman villains, even though that’s all [property of] DC, someone still has to go out and find it, you know, the guy that wrote Gentleman Ghost into the issue where he first shows up. I definitely kept a lot of coordinators, producers, and lawyers busy on this movie. I’m sure I drove people crazy."

Now there may have been a financial component but we're unlikely to find details of those anywhere.

In some case, they actually had involvement from the creators/rights-holders like JK Rowling.

McKay: "We used Dumbledore in The LEGO Movie as a quick one-off joke, but Voldemort obviously plays a bigger role in this movie, and J.K. Rowling obviously cares very deeply about these characters, so we absolutely had to run stuff by her."