What was the first instance of a movie reboot?

This originates in the world of comics, and there have been many "reboots" of comics turned into TV series and/or TV series turned into films.

But assuming your definition is restricted to movies rebooted from other movies only, Wikipedia tells us Godzilla is the earliest. First made in 1954, and rebooted at least in 1984 and 2000.

As new directors came in over the years, the style and story changed considerably. (See the question on this site asking if the monster Godzilla is good or evil.)

The earliest instance of a movie re-boot I can find is that of Tarzan. The earliest film was made in 1918. The same wiki link also shows us:

With the advent of talking pictures, a popular Tarzan movie franchise was developed, which lasted from the 1930s through the 1960s. Starting with Tarzan the Ape Man in 1932 through twelve films until 1948, the franchise was anchored by former Olympic swimmer Johnny Weissmuller in the title role. Weissmuller and his immediate successors were enjoined to portray the ape-man as a pidgin-speaking noble savage, in marked contrast to the cultured aristocrat of Burroughs's novels.

This "me Tarzan, you Jane" characterization of Tarzan persisted until the late 1950s, when producer Sy Weintraub, having bought the film rights from producer Sol Lesser, produced Tarzan's Greatest Adventure followed by eight other films and a television series. The Weintraub productions portray a Tarzan that is closer to Edgar Rice Burroughs' original concept in the novels: a jungle lord who speaks grammatical English and is well educated and familiar with civilization. Most Tarzan films made before the mid-fifties were black-and-white films shot on studio sets, with stock jungle footage edited in. The Weintraub productions from 1959 on were shot in foreign locations and were in color.

While re-imaginings of stories have been told over and over in film and other media since long before film began, the first instance of the use of the term "reboot" was 'The Incredible Hulk' in 2008, which came out only five years after the previous movie 'Hulk'.

The studio felt that they needed a word to help describe what they were doing with the movie because it was not a sequel, nor were they starting over with another origin story. Instead, 'The Incredible Hulk' started with a quick gloss over the origins of the Hulk and then jumped ahead four years, leaving it open-ended as to if the events from the previous movie were considered to have happened or not in this story arc (as we now know, they did not).

It's also worth noting that there were questions at the time as to why the movie was being remade so close to the previous film, and no one could call a movie with a full recast a "sequel." This put Marvel in a position where they needed to create a marketing spin, and thus the term "reboot" was born. At this time, they didn't want to tip their hand to the idea that they were going to start a series of Marvel movies that all had plot tie-ins to each other, nor had they constructed any long term plan for such a series (the series we now know as the Marvel Cinematic Universe, beginning with 2008's Iron Man).

As pointed out above, the term was needed because of the proximity of the two movies. In 2005 when Christopher Nolan's new vision of Batman surfaced with 'Batman Begins', it was referred to generally as a "re-imagining of the Batman universe." It has since been retroactively deemed a "reboot," but the time gap as well as the complete change of tone between Nolan's Batman series and the 1989-1997 Batman series makes it unnecessary, particularly when considering how long ago the first series began.

Ultimately, the term has given us a much needed word for movies that fall into a certain category, we just never knew we needed the word until 2008. The word has since gained a more concrete definition: to discard all continuity in an established series in order to recreate its characters, timeline and backstory from the beginning. It's overuse in the comic book genre has generated some push back from fans when the term gets retroactively applied to media older than the term itself, which may also have to do with the vague and ever-changing definition.

It may not be the first, but the earliest film reboot I know of is "The Maltese Falcon". The first film adaptation was in 1931, and the second (under the title "Satan Met A Lady") was in 1936.