Google indexing zero for 50k products - is Cloudflare to blame?

ryanseo

I've been wrestling with a strange indexing issue on a WooCommerce site with around 50,000 products. Everything looks fine on paper - robots.txt is clean, no noindex tags, valid sitemap submitted to Search Console, no crawl errors. Googlebot is hitting the server regularly (confirmed in logs), pages are returning 200 OK, and images are being crawled. Yet the Google index has exactly zero pages from this site. Bing, on the other hand, has indexed about 11,000 pages without issue. So the site is clearly crawlable and indexable - just not by Google.

My Cloudflare WAF has a rule that blocks a long list of bots - Ahrefs, Semrush, MJ12, GPTBot, Baidu, Yandex, and so on - but I've explicitly allowed Googlebot. Still, something is off. Google crawls but essentially ignores the HTML pages, focusing mostly on static assets. URL discovery in Search Console is nearly flat.

A colleague suggested it might be a content quality and authority problem rather than a technical one. They asked whether I had unique product descriptions or just manufacturer boilerplate, whether there are user reviews with real human input, editorial content, an about page with actual people, or if I'm using AI-generated slop. Even without AI, if the site feels faceless and generic, Google might categorise it as low-effort content.

To be honest, I've poured a lot of effort into SEO for each product - better titles, short descriptions, longer descriptions, detailed specs. Yet after five months, I'm seeing 11k crawled but not indexed, 12k detected but not indexed. Scan stats show 99% OK, but the breakdown is weird: 100% 'update', only 1% discovered. Photos account for 57% of crawl activity, mobile 27%, desktop 13%. I've added meta descriptions, reduced TTFB from 2.6s to 0.9s, still nothing. Bing indexed everything immediately even before those optimisations. Honestly starting to lose hope.

analyticsjunkie

oh mate, been there. first thing-check if GSC says "Crawled - currently not indexed". if yes, this isn't a technical wall, it's a content quality door they're not opening.

you got unique product descriptions or are you recycling the same manufacturer boilerplate everyone and their dog uses? user reviews with actual human souls, or just star ratings scraped from somewhere? Any editorial content-blog posts, guides, anything a real person wrote?

Do people actually link to you? share your stuff? Recommend you? or is your site an island of silence?

And the big one-do you have an About page with real humans, names, photos? if there's no visible pulse, Google assumes it's automated slop. Even if you're not using AI, if it reads like boilerplate, you're categorised as low-level content. Sucks, but that's the game

nightowl

It's surprising how often a single boolean mistake can cut off the entire search traffic funnel. This is like having a bouncer at the door who checks for any bot ID and lets them in if they match any of the patterns - but that means Googlebot gets blocked because it also matches one of those user agents? Actually, the issue is you're using OR between the conditions. That means Googlebot will be blocked if it matches any of those three agents, which it probably does. What you want is AND to only block traffic that matches all three, which is essentially impossible. Googlebot alone won't hit all of them. Swap OR for AND and you'll stop blocking the very crawler you need for indexing. I'm not at my desk now, I'll take a proper look in about ten minutes - could be I'm misreading your syntax, but that's my immediate read.

storyweaver

Right, so you've got 50k products and zero indexed pages. that's not a glitch - that's a flag the size of a billboard.

Did these 50k entries all land at once? Like, did someone just mash the "import from CSV" button and hope for the best? Are we talking auto-generated descriptions, maybe scraped from a supplier with zero added value? because if you also pumped out a ton of programmatic blog posts at the same time, Google's going to smell scaled content from a mile away. Low information gain + scale = spam signal. Bing's threshold is way lower, which is why you might see some traction there.

Cloudflare can play tricks on you too. A JS challenge page can return a 200 OK in your server logs, but Googlebot sees the challenge, not your actual content. that's a classic "looks fine in logs, dead in search" scenario.

What does the live test in Search Console show? look at the actual screenshot and raw HTML. Is the number under "discovered - currently not indexed" suspiciously close to your total page count?

Is the homepage even indexed? What's the domain history - was it previously a dodgy PBN or parked page? And please tell me you didn't build a bunch of sketchy anchor text links to these product pages.

one more: can Googlebot see the page content without JavaScript? How heavy are these pages? What's the average TTFB looking like?

I'd start by sorting that content quality and crawl budget mess before blaming Cloudflare, but yeah, the CF challenge thing is a known pain.

sophiasaas

Without the URL it's tough to diagnose, but I've seen Cloudflare firewall rules block crawlers before - worth checking your WAF logs

neonlights

First thing: what's your Google Search Console telling you? That's your single source of truth here. If GSC shows pages submitted but zero indexed, you've got a crawl or rendering problem. if they're not even submitted, that's a whole different fire to put out.

Before you go blaming Cloudflare, check the Index Coverage report. 90% of the time when I see this, the WAF is blocking Googlebot's IP ranges or the site is loading fine for humans but failing the render test. Had a client with 30k product pages - turned out a rate limiting rule was catching Googlebot. Fixed that one rule, indexing recovered in 10 days.

What's your "Discovered - currently not indexed" count looking like?

ryanseo

You've done a lot of the right things on the content side, and it's frustrating as hell when Google just sits on its hands. I've been through similar scaling builds where the crawl budget gets choked off by infrastructure, and the fact Bing had you indexed in days tells me the problem is almost certainly server-side or WAF-related, not content quality.

A few things to look at in order:

Crawl log analysis - Do you see Googlebot hitting a 403 or 503 consistently? Cloudflare's "Under Attack" mode can block bots even if you think it's off. Check the firewall analytics for any rules targeting Googlebot or Google User-Agent. sometimes a security rule catches all unknown UAs and Google falls into that bucket.
robots.txt - Double check it's not blocking CSS/JS. With 12k detected and 11k crawled but not indexed, that disconnect means Google is hitting the pages, reading them, but then deciding they're either low-quality or blocked. If you're using Cloudflare's rocket loader or a JS-heavy theme, the rendered HTML might be empty.
Canonical and noindex tags - Run a site-wide crawl with Screaming Frog or similar. Make sure no product pages have a <meta name="robots" content="noindex"> injected by the CMS. happens more than people admit with layered caching plugins.
Core Web Vitals - TTFB dropping from 2.6 to 0.9 is solid, but check the rest of the waterfall. if images are huge (57% of your scans being photos suggests a lot of resources), Google might decide the pages are too slow for mobile indexing. compress aggressively, lazy load, and make sure the LCP element isn't an image above the fold that takes forever.
Indexing API - For a 50k product store, you can't rely on natural discovery. Use the Indexing API on new or updated products, but only if they're truly unique content. Google has a limit of 200 requests per day per property, so focus on the highest-margin product pages.

I've had sites where the WAF was returning a soft 403 (a 200 status but the page body was a blank or a captcha wall for bots). Google sees a 200, stores the URL, but can't render anything useful. Check the crawl-status in GSC for any signs of "Blocked due to access forbidden" or "Other" errors.

One more thing - the "crawled but not indexed" bucket is a sign of thin or duplicate content. even with attribute specs and long descriptions, if the product page is largely the same template across thousands of items, Google may treat them as low value. Consider adding unique editorial content per category or tier of product, not just per item.

Don't lose hope yet - you've already fixed the server response time, which is the hardest part to control. The indexing issue usually comes down to one or two misconfigured rules in the middle. Bing indexing fast just proves the content is good enough. Google's just being a prick about how it receives it

emmareach

The first point in that reply is spot-on - if Google starts seeing patterns it doesn't like, it'll fingerprint your URLs and stop indexing them altogether.

From what you've said about "generating a better title," I'm guessing these are AI-generated product pages? If so, that's a huge red flag for quality scoring. Google tends to lump that content into the lowest tier, and once you're there, getting back into the index is an uphill battle.

Without more details it's hard to be certain, but based on past experience with similar setups, that's the most likely culprit.

nightowl

That's the problem right there - a 200 status on a captcha page tricks Googlebot into thinking it's actually crawled a valid URL. Instead of seeing your product content, it's served a gatekeeper page. So every one of your 50k products is effectively a soft 404 in Google's eyes. Cloudflare WAF rules need to return a 403 or 503 for challenge responses that hit bots, not a 200. It's a classic case of the platform treating all traffic the same way, but search engines require different behaviour - like Amazon's A9 knowing the difference between a user hitting captcha and a crawler hitting an indexable page.

storyweaver

oh, sweetie, the Cloudflare isn't your problem. You've just built a giant red flag and waved it right in Google's face. Fifty thousand products, a low-trust domain, and every single page plastered with LLM-generated fluff? that's not SEO-that's spam on an industrial scale. google's spam policies literally call that "scaled content abuse." You've basically handed them a smoking gun and said "please ignore me."

Honestly, starting from scratch might be cheaper at this point. I'd nuke 99% of those products, get the homepage and main category pages indexed, then drip-feed the core ones back in while watching what Google does. Anything else is just throwing good money after bad.

ryanseo

Had almost the same problem with a client's subdomain. Even if the root domain has authority from 2006, Google often treats a subdomain as a separate entity-especially if it's using Cloudflare.

A few things to check methodically:

Server logs - Are Googlebot requests being blocked? Cloudflare's WAF rules (especially under "Bot Fight Mode" or custom firewall rules) frequently return 403s for product page crawls.
Crawl budget - With 50k products on a subdomain, Google might deprioritise it if internal linking is weak. Check that your sitemaps are pointing to the subdomain URLs and not the main domain.
Response headers - Look for X-Robots-Tag or noindex directives that could be injected by the WAF. I've seen misconfigured rules strip away index status.
Robots.txt - Is the subdomain's robots.txt blocking anything? Some setups inherit the root domain's file, which can cause issues.

The fact Bing indexed immediately rules out most duplicate-content concerns-it's almost certainly a crawl/access barrier. Run a live URL test in GSC for a product page and check the server response. If it shows "blocked by robots.txt" or "soft 404", Cloudflare is almost certainly the culprit.

nightowl

cloudflare's probably not the issue. Check robots.txt and sitemaps instead.

ryanseo

Honestly, zero indexed pages on a 50k product catalogue is brutal. I've been there on a few programmatic builds this size, and it's rarely one single thing - more a cascade of issues that compound.

First off, Cloudflare/WAF can absolutely block Googlebot if you've got aggressive rules. Seen it happen when people enable "I'm Under Attack" mode or set a challenge for certain user-agents without whitelisting Googlebot's IP ranges. Check your WAF logs for any 403s or challenge pages served to Googlebot's IPs. Also worth verifying that Cloudflare's "Automatic HTTPS Rewrites" or "Rocket Loader" aren't stripping critical crawl signals.

Beyond that, for a catalogue that large, crawl budget becomes a real bottleneck. If your site architecture is flat or you're throwing 50k URLs at Google without proper prioritisation, they'll only crawl a fraction. Things I'd methodically check:

Internal linking structure - Are your key product pages reachable within 2-3 clicks from the homepage? Use a tool to map link depth.
XML sitemaps - Split into multiple sitemaps (max 50k per file) and ensure they reference only indexable, canonical URLs. No filtered/faceted variants.
Server response times - If pages are slow (3s+), Google will crawl fewer. Cloudflare can help cache, but dynamic product pages might still hit origin.
Log file analysis - The only way to know exactly what Googlebot is seeing. Look for patterns: are they hitting 404s, redirect loops, or being blocked by robots.txt rules?

Also, check your robots.txt hasn't accidentally disallowed /product/ or a similar directory. Seen that more than once after a Cloudflare migration.

It's a slog, but start with the log files and WAF logs - those will tell you if Googlebot is even getting through. If they are, then it's likely a depth/crawl budget issue that needs structural changes and better internal linking strategy. Happy to dig into specifics if you share more on what your current site architecture looks like.