Large gravel, as you may be finding, hides a lot of dirt and detritus. Fry can sometimes hide in the spaces and I have even spawn fish on purpose over such gravel because eggs also can be beyond the reach of adults.

What kind of gravel do you have? Is it a natural type, which comes in several sizes.

I have seen some rather attractive tanks which actually feature what is essentially the same gravel, but in different sizes.

I wonder if you would mind mixing some of the same gravel, but with a smaller grain gradually worked in with your larger gravel. Perhaps just before doing a light gravel vacuuming, you could sprinkle some of the small stuff on the surface and then gravel vac. That way the "live" gravel isn't suffocated. The new stuff is started as a home for beneficial bacteria.

It might be a couple of sprinklings before you could see a difference. You might want to try that just in a corner of the tank to see if you like that combination or take a little gravel out of the aquarium and mix the two in a jar.

I may do something of this sort myself, having locally bought two 50 lb. bags of a gravel which is a beautiful red flint. For an example see http://jonahsaquarium.com/tankdeco.htm

Unfortunately it was a larger size that I need. Worse, I didn't open a bag for a couple of months.  I stopped by Glenwood Pets and Hobbies and explained to their manager/fish man, Paul. He was most willing to trade them back and even offered to take back the bag I opened. Turns out that his mostly cichlid keeping clientele like the big stuff and he has more demand for that. I may have done him a favor. ;)

I felt funny about giving him the opened, slightly used stuff, but cheerfully traded untouched the full bag. I'm going to rinse out some of the small stuff and mix it with the large gravel and see how it looks.  I don't know that with gravel vacuuming, if the larger stuff will work it's way up.

If you are not interested in this, that is no problem for me. Maybe I can try this in a week or so and let you know what it looks like. And/ or you can try that mix in a jar or bowl. :)

It does sound like a pain in the neck partially restarting with brand new gravel. Can you do that in a modestly populated aquarium, with a well-seasoned filter and plants?