Why does the Machine remember everything?

The Machine was programmed to erase its memory at midnight everyday. But it has been evolving, it is always evolving. Evident by the fact it is talking to Root as an analog interface.

And its probably hiring people as seen in the ending scene, where a woman uses the same speech Finch used to hire Reese.

In Season 2 Eps 21 - Zero Day, Finch learns about how The Machine has created a fake identity and a company who hires people to re-enter data after midnight, essentially re-enter its memories.

In Season 5 Eps 2, when The Machine got access to the feeds and internet, it then deemed Finch, Root, Reese to be a treat, before that it was running ok, One can only guess that since The Machine did not have access to the history feed.

You also have to notice that The Machine makes judgement after visiting the past records/feeds/videos/data (whatever you wanna call it). And those records are not memories and hence are not deleted. So those records are probably saved somewhere.

Coming back to present, since The Machine have no number system, it probably read all the records at once and hence all the times Finch killed it.

Now you maybe asking where are all those videos saved? I have no clue. But it must be saved somewhere, as The Machine makes it judgement after watching the history.

If everything the machine learned during a day were erased at midnight, the machine would be unable to learn over time. One episode shows Finch trying to teach chess to the machine, so we can assume he wanted the machine to learn new things over time. The "erase all data at midnight" rule applies only to the persons of interest the machine learns about but not to metadata, such as the process the machine uses to make its decisions (such as how to locate persons of interest).

Imagine you having access to hundreds of books and the ability to read and understand them all in a matter of minutes. But, here is a cliche, you forget everything you have read.
This might be the case of the machine, it had the access to the feeds and internet and analysed everything daily(after its "erase all data at midnight" rule)
In S05E02 the machine had access to the feed of past and real time

but was unable to differentiate the timeline. For the machine it all felt like its happening at the same time

this can be backed up

as root says "She's falling through time.Probably a serial processing issue." Which means the machines parallel processing was working fine not the serial which explains when Finch asks "What day is it?" which is replied as R

Then,

What Finch replies solves everything "R, the mathematical set of all real numbers. Today is every day to you. I killed you 42 times. And you are constantly reliving those 42 deaths." Thus, the machines parallel processing was going through those days again and again(imagine reading same set of books reading again and again)