If you create new timelines when you change the past, the Captain must have traveled back to this one after growing old.

By universe rules,

you cannot alter current timeline (that is why they couldn't kill Thanos as a baby).

So, when Captain stayed with Peggy, he created and lived in an alternate timeline. Presumably, everything happened similarly to this one.

This is supported by this interview.

Cap did, in fact, create an alternate reality, one where he lived his life with Peggy.

Q: Peggy Carter was probably already married and in her mid-40s in 1970, in that case, what year was it that Captain America went back to dance with her?

A: We can’t answer it, for now, this is a story that happened in an alternate reality. [...] The time travel in this movie created an alternate reality. He lived a completely different life in that world. [...] If you go back to past, you simply created a new reality. The characters in this movie created new timeline when they went back to the past, but it had no effect to the prime universe. What happened in the past 22 movies was still canon.

And at some point, he had his time-line's Pym, Scott, Stark, and Banner to bring him back to this one. Or maybe he used some other method unknown to us viewers.

In the original timeline, the Tesseract was lost in the sea, Steve stuck in ice, Howard searches for Steve and finds the Tesseract and gives it to SHIELD for research.

In Endgame, Tony takes Tesseract from SHIELD, so Steve already went under ice. So the current Steve goes back to this moment to return the Tesseract.

There are two at this moment but one stays with Peggy and one is stuck in ice.

When Fury finds Cap in the original timeline, that sequence of events continue as it is. At this moment, there is a young Steve and very much older Steve and this older Steve returns to the present. Thus completing his timeline.

There is only one Steve.

We can expect he approached Peggy sometime around the late 1940s, and said "look, here's what happened. We can't mess with the timeline, so if we want to be together, we must be discreet about it."

Hence the unremarkable, nondescript little 1940s house.

And Peggy allowing it to be believed she made her life without him, and that this husband of hers just merely resembles a 12-years-older Steve. (since Steve has aged normally from when he was unfrozen in 2011, to 2023 when he returned the Infinity Stones).

So from 1947? until ??? there are two Steve Rogers; one frozen in the ice, one living a quiet life with Agent Carter.

In 1970, there are three: One frozen in ice, one living a quiet life, and one raiding the Army base for the Tesseract/Pym particles.

The movie doesn't give any details about what Cap did, so I don't think anyone can give an official answer yet, but I think the implication is that he went back to Peggy at some point after the original him was already frozen. They lived a (presumably) quiet life together, which probably would have required moving away from people who recognized him. That would mean there were two Caps after that point, but since the second one never did anything significant it didn't affect things. However, it's unclear how he avoided messing with the parts related to Peggy living without him; it's just not shown.