What's the background story with Joe Buck and Annie?

In the film Midnight Cowboy, the background story of would-be hustler Joe Buck is given in a series of flashbacks, many of which feature a young woman named Annie. Because the flashbacks are so brief and disjointed, it's difficult to piece together a coherent story. We repeatedly see Joe and Annie having sex, and we repeatedly hear statements or voice-overs from various women (including Annie) telling Joe that he's the best or that he's the "only one". We also see scenes of Joe with a group of men, and scenes with Joe or Annie running with or from a group of men, or being restrained by them. In some scenes the police are involved.

What exactly is Joe's relationship to Annie, and what happened between them? Was she his girlfriend? Why is she seen with the police, or running from the group of men? Was she the victim of a rape or other assault, and if so, was Joe participating in it or defending her from it? Is Annie's statement that Joe is the "only one" meant to be an accusation to the police that he assaulted her? (And if Joe really was a violent rapist, why is he so kind, forgiving, and helpful to everyone in New York? The only time he's shown using violence or taking advantage of others is in what he believes to be a life-and-death matter to save a friend.)