Why does Keller show up unprepared at the end?

After an exhausting ordeal he's been through and after losing almost all hope, he hears "You were there" from Joy. Moments later he realizes where was the only place where he could've been heard (we see that clearly on his face), and it dawns on him where his daughter is (or at least was). Also, in his mind, Alex is still the guilty one.

So, I see his behavior as a consequence of two things:

1. Since he believes Alex is the kidnaper, he sees Holy as only protecting Alex by not revealing his secret or, at best, by taking care of the kidnapped children. Ergo, she doesn't really pose a threat.

   I think he confirms this when he comes in her house and, obviously trying very hard to restrain himself, says

   I don't want to have to hurt you. I know they were here.

   This is said in a peaceful, but threatening manner. Also, it is not "(I know) you took them", but merely "they were here".

2. The whole realization came as a shock, after days of fruitless search and Alex' torture. He's not in his right mind, and he didn't think it all through.

Also, he wasn't completely unprepared. After Holy dumps him in that hole under the old car, she goes back to her house and starts going through his tool bag (the one he had in front of her when he said he doesn't want to have to hurt her), where she finds his gun. So, he was routinely prepared, but not for the "Holy is the only culprit here, not Alex who he had safely away", which made her better prepared.

Mostly it is due to the fact that he has just spent a large portion of time torturing an innocent man. At this point, he is mentally exhausted. As you say, it does come across more like he is more concerned with finding out who really took his daughter, than actually finding his daughter. So by this time he is drained and spent, and he is not looking to use force to intimidate the answer out of her.

Because his actions ended up being questionable (as opposed to being justified), he may believe that somehow coming to Holly with just the desire for the truth will be enough. This is why he ends up in a hole, and why although we are left thinking he "might" be rescued, we are not actually shown this in the film. For what he did, he may even believe he has deserved his fate. This belief would also reinforce his desire NOT to bring a gun to the confrontation.