The Solution? Purge your Render Appearance Property Sets (make sure you know the implications of this).
After purging these, the lag went away. However, and it is a big 'however' - this resulted in some of our renders no longer working correctly.
If I first went into the few materials that used property sets and set them to 'By Property Set' and not 'Independent', Revit did not purge the necessary data and the renders were still ok.