Does the FBI keep track of people's library records?

Basically, it's fictional (at least legally)

Forty-eight states and the District of Columbia have laws protecting the confidentiality of library records. Two states, Kentucky and Hawaii, have attorney general's opinions protecting library users' privacy. The language of these provisions vary from state to state. The majority of these laws declare that a library user's records and information are confidential, and not subject to disclosure, unless certain conditions are met, such as a user's consent or the service of a court order.

Source

BUT...there was such a program.

As early as 1973, the FBI was running a program aimed at securing information about reading habits of many library users; this program was ultimately called the "Library Awareness Program". The Library Awareness Program was designed as a counterintelligence effort that would provide information to the FBI including the names and reading habits of users of many different libraries. The FBI was particularly interested in learning this type of information about foreign diplomats or their agents. It is clear that librarians and the public were unaware of this program until its existence was made public in an article published September 18, 1987 in the New York Times.

Wikipedia

Via the Patriot Act, it is possible to request library records. However, librarians found this offensive, so they responded by only keeping the absolute minimum possible records. Therefore, the FBI may be able to see what you have checked out at the moment, but not what you have checked out in the past, because the library erased all records of those checkouts as soon as you returned them. The specifics may vary by library, but this is broad policy.

(I'll note that the answer by Paulie_D describes the state-level laws, but federal law supersedes state law. I'll also note that this law was made after Se7en released, so it couldn't have been used in the film, but you're asking about if it's possible in real life.)

Example policy

The US government monitors all kinds of internet traffic. To say that they listen to private lines (whether internet or phone) but they do not listen to a publicly funded institution is a little ridiculous.

After 9/11 there were many monitoring agencies that picked up on government activity specifically on libraries. All they have to do is save the information, as if a library deletes it, the info is still stored. There is also no reason that our public libraries databases (internet connectivity) are Fort Knox-like.

So to assume that the FBI/CIA has discontinued doing this because the Patriotic Act or some other act is RIDICULOUS. There are two reasons why we don't hear about it more now.

1. Since it is "illegal" (saying a govt agency unilaterally doing this is illegal is funny it is akin to making your dad eat his vegetables) they simply don't mention it anymore.

2. Since the knowledge shift for things that may help "bad" guys has shifted from books to the internet there is less of an emphasis on this.

However... There have been several stories about how people were tracked because of internet activity at libraries. If you read about a story there are 50 more that happened that nobody heard about.

No, this is not realistic, not in any library staffed by librarians who have degrees from programs accredited by the American Library Association (ALA) and/or who adhere to the ALA's Code of Ethics, which should be all of them in the U.S.

Patron privacy and confidentiality are near the top of the list of important things librarians care about, just behind access for all and anti-censorship.

Point #3 of the ALA Code of Ethics states:

We protect each library user's right to privacy and confidentiality with respect to information sought or received and resources consulted, borrowed, acquired or transmitted.

Item VII of the Library Bill of Rights states:

All people, regardless of origin, age, background, or views, possess a right to privacy and confidentiality in their library use. Libraries should advocate for, educate about, and protect people’s privacy, safeguarding all library use data, including personally identifiable information.

The simplest way to ensure that patrons' borrowing records are private and secure is not to retain them at all, which is, in fact, how things work at the libraries I patronize.

The bad guys can't get what doesn't exist.