I don't think this is very difficult to understand. When you see YouTube or any service where you're expecting ads to see, you won't feel your privacy being intruded. But in cases of cold mailing, cold calling etc., even if the data has been acquired in a 'legal' way, (often when the data is sold and exchanged), the user just clicks on I understand the Terms and Conditions, which inside does say that it's permissible for the company to use this data for third party services, but the user in real terms hasn't signed up ever for any such third party service.
That's why you feel annoyed when someone randomly calls you selling games, insurance, courses etc. It's the same with cold mailing, the difference although being that the mailing can be handled much more strictly, and infact is handled strictly by Google and others, because for them the top most priority is user trust because that's what on their whole eco system is dependent on, and hence every decision is guided by this basic principle: trust.
Yes they're setting rules, yes cold mailing is getting more difficult, and trust me the frustration is real, but I personally cannot blame Google or anyone if they're taking it so seriously.
Hope this helps