Why aren't Kaylie and Tim packing the things and leaving the house since the experiment is impossible to complete?

In this movie, we see a a sister waiting for her brother to return from incarceration, to be able to perform a video documented experiment and destroy an antique mirror which is supposed to be the reason for more than 25 deaths.

Kaylie Russell: I found it!
Tim Russell: What do you mean? Kaylie Russell: We only have few days.
Tim Russell: A few days for what?
Kaylie Russell: To keep our promise. And kill it.

From this conversation its clear that, destroying the mirror and facing the entity and vanquishing it was not just a temporary whim but a well planned design and Kaylie was waiting for Tim to return.

Tim tells Kaylie their father was having an affair and her memories regarding the mirror are fabricated; that it was a bad marriage and that was it. Kaylie argues she went through everything in the estate and there was no proof of an affair. The two continue arguing until Tim gets frustrated and lets the dog out of its cage and out of the house. He points out that the plants are still alive and the only one endangering the dog is Kaylie. He then begs Kaylie to leave the house with him, saying they can go somewhere and deal with everything together. Kaylie agrees to disarm the trap and take the cameras down, but just as they are about to do so Kaylie sees that the plants in the house are dead and the cameras have been turned away from the mirror. They replay the tape and see that during their argument they were subconsciously moving the video cameras. Kaylie also knows that the temperature has gone up over five degrees but the alarms have not gone off. Tim goes to call his doctor but Kaylie stops him and says to make any calls outside, as the mirror can mimic voices on the phone so long as they are within its range. IMDB Synopsis

Ans finally, this was the device, which was planned to strike and destroy the mirror once and for all and the entity too.

To answer your question: The fact that the camera positions had changed in their absence, the plants dead, the room temperature rise: all of these were the very reason why they had to go on and they could not just leave. It was apparently the mirror that destroyed the childhood of these two individuals as is shown in the movie. The reason why they could not leave was that, everything was real. The changes were done by them but they could not remember doing it. The mirror, the entity was toying with their senses and causing them to hallucinate like seeing a broken bulb, in hand after biting off an apple. They had to try their best to destroy it and to find answers. Being through such a trauma of knowing their father killed their mother, they just could not let it go and wanted a closure. More than anything, the brother and the sister wanted to ascertain that along with those 25 deaths, their mother was not ACTUALLY murdered by their father and the mirror had something to do with it. Hence the video taping.

NB: they never for once came to the conclusion that it was impossible to complete the experiment.