Hi,
First of all, FB recommends using a hybrid tracking method (web events + server events with deduplication). Web events also collect microdata, this can probably affect audience collection as well. I would recommend trying to use tracking as FB recommends and see how it affects your problem.
The second is to increase Match Quality you need to send user data where it is available to you. It’s something that directly affects the effectiveness of audience collection. IP address, User Agent and fbp are basic parameters so if you want to collect audiences more efficiently, you need to add user data that you have available.
Here you can find all the possible user parameters you can send and their formats: Paramètres - API Conversions - Documentation - Meta for Developers