This one is a bit difficult... All gallery plugins claim they have the fastest loading gallery, and all will feed you with tests to support their claim.
From my humble experience, what could be considered needs to fit a couple of conditions:
One of the plugins fitting the above is Meow Gallery.
The following page has a 16 pictures gallery (ranging from 1 to 2mb each). And yet, the whole page loads in 180ms
Yes, there are a couple of trick to get that kind of loading time, but still, using Meow Gallery helped with that too.
Meow sits on top of the native WP gallery with minimal added code + a pretty good lazy loading system.
I picked Meow for it's simplicity and performance, and also for a couple of additional plugins part of the Meow Apps ecosystem (namely the Retina plugin and the Lightbox plugin).
Other galleries have similar features. But non has that much granular features enabling and a low scripts/style footprint. It gives you the minimum you need to get the job done.