How did the plan work out at the end?

On Wikipedia's page for the movie plot it says:

Malthus' death erased himself and his murder of Abe from the future.

So you can chalk that one off as a bad writing / writers not understanding how time travel works / glossing over these details to make a happy ending.

If Abe doesn't get killed, Jake (in that timeline) will not travel to meet Miss Peregrine (he only goes to Wales after Dr. Golan helped him to convince his parents) therefore possibly not killing Malthus, breaking the whole time travel in the movie.

The ending does not make sense. At least to me. Here is what Emma tells Jake while they're aboard the ship on the way to Blackpool (the location of Miss Avocet's January 2016 loop):

Jake, you know what this means don't you? ... We have to get to our own time but you don't. You can continue to live in 2016... If we kill Barron, he'll be gone but time will carry on. He won't go to Florida, Abe won't die, and you can go home."

At least to me, it doesn't make logical sense that killing Barron would save the life of Jake's grandfather Abe. Abe died in July 2016, so theoretically killing a version of Barron from before then would prevent Abe's death. But that's not what happens; in Blackpool Barron speaks about Abe's death:

I tracked Abe Portman for months. All I needed was a few minutes to get the location of Miss Peregrine's loop from him. But no, Mathus couldn't wait. And instead, I had to masquerade as a psychiatrist for three weeks in Florida.

So Barron has already participated in the Florida events. So it does not make logical sense that killing Barron will mean that "he won't go to Florida [and] Abe won't die".

The movie logic seems to be something like "If you die in a loop then both your past and your future are completely erased from the timeline." I think that's the meaning of the statement "he'll be gone but time will carry on". But it seems impossible to come up with a model of time travel where that would make sense.

No, time was linear for the main characters. Baron's associate, the hollowgast, killed Abe, which led Jake to the island, which allowed Baron to locate the loop, which led to the chase and the London loop. The London loop was 6 months in the past, which is what allowed Jake to visit his grandparent. Only visit, not save. See https://scifi.stackexchange.com/questions/142842/how-did-this-character-survive/142878 for additional comments.