How to change settings for Win10 File Explorer’s “Frequent Folders”

I use the File Explorer’s Quick Access “Frequent Folders” section a lot but it seems to take a while before a folder will appear there. Is there a way to change the setting so folders will appear sooner? I know how to pin folders but that’s not what I want to do.