What is the backstory behind the grandmother and her missing sister?

It's in the books. I'm going to just paste this from the Coraline Wiki, as there's no sense re-writing a good explanation.

Before the Pink Palace was separated to apartments, Mrs. Lovat used to live there with her twin sister. But in 1952, her sister mysteriously vanished one day, and Mrs. Lovat believed she was "stolen." Her grandson, on the other hand, believes that she simply ran away. It is revealed that Mrs. Lovat's missing twin sister Tasia ran away and was stolen, by the The Beldam (Other Mother).

Believing that someone "stole" her sister and possibly hearing about two other kids disappearing before her, Mrs. Lovat moved out of her childhood home and divided it into three apartments. Afraid of the Beldam claiming another child, she does not allow Wybie to enter it, nor allow any tenants with children to rent the apartments, until when The Jones' Family was allowed to reside there.

Even when her twin sister went missing, Mrs. Lovat kept the doll that used to belong to her sister in a trunk, perhaps for sentimental purposes. The doll, which used to look like her sister, was altered by the Beldam to look like Coraline, as seen in the beginning of the film. At the end of the film, Coraline invites Mrs. Lovat and Wybie to the garden party, where Coraline tells Mrs. Lovat what really happened to her missing twin sister.