Absolutely, you can overdo it - but let's be real, the chances that those junk clicks suddenly turn into gold are next to nil. One quality click that converts? Worth fifty that don't? Please.
I had a campaign where I was playing whack-a-mole trying to block terms centred on a product I didn't sell. I sold something adjacent for people who specifically didn't want that product. In the end I gave up and nuked the whole batch. Maybe I lost a handful of sales. But the pile of crap I'd never stop? Worth it just to sleep better.
Here's the thing people miss: even when you block those, Google just funnels more rubbish your way. It's baked into their business model. They make their share price go up by keeping that line of junk flowing - your negative keywords list is just a speed bump, not a wall.