Avoid flutter Consumer widget children rebuild

I am developing an app allowing to download maps of regions and show either on a map (using flutter_map) or in a list view. While downloading a map, a progress bar must be displayed on the list view, and the map must be refreshed only once the download is done. I'd also like to avoid reloading all the maps when one is downloading / downloaded / deleted.

To achieve this currently I wrote a ChangeNotifier handling the maps download & deletion, and Consumers in the list & map views. It works well but whenever I call notifyListeners() in the ChangeNotifier, all the maps are reloaded, which causes a blink on the map view.

Here's the map view consumer:

Consumer<MapsProvider>(key: const ValueKey<String>("maps"), builder: (context, MapsProvider mapsProvider, child) {  return _loadMaps();})

The _loadMaps() returns a list of Widgets, each having a ValueKey<String> uniquely identifying each map.

Is there a way to only reload the widgets that effectively changed?