built 900 pages in a weekend but indexing tanked

lucasvid

Spent a weekend diving into a rabbit hole of niche tool directories. Thought it'd be fun to generate my own from a public dataset instead of hand-curating every entry. Scraped ₉₀₀ dev tools and AI products, cleaned them up, normalised categories and tags. Simple Next.js script pumped out static pages in ten minutes. Each tool got its own page, plus category pages, internal links between similar ones. Looked solid.

Submitted the sitemap to Google Search Console, sat back, and expected discovery to just... happen. Two weeks later, only about 120 URLs were indexed. The rest sat in "discovered, currently not indexed" purgatory. That retention curve looked brutal.

Tried manually requesting indexing in GSC - could only do ₁₀-15 a day. Added stronger internal links, pushed updated sitemaps, tried some WordPress pinging tricks. Some pages crawled after 10-14 days, but most lagged behind. Eventually wired a queue to watch the sitemap and submit URLs via APIs. That script needed constant babysitting, so I tested IndexMeNow and then IndexerHub just to handle the submission layer without losing my mind.

The real lesson wasn't about the tool - it was that publishing hundreds of pages is easy, but discovery and indexing becomes its own system once you cross a few hundred URLs. Volume doesn't fix discoverability. I'm curious how other builders here handle that when shipping directories or programmatic SEO projects. Do you just wait it out or actually build an indexing workflow? Someone in the thread said 900 pages indexed but did any actually bring traffic - that's the gap most people miss. Hits hard.

wordweaver

900 pages in the index, but let me guess - sweet FA in terms of actual visits? that's the trap everyone walks into. more pages doesn't magically equal more eyeballs. Discoverability is a completely different beast from volume

cpaoptimizer

Ran into the exact same wall myself with a dataset of about 1.5k open-source tools. Throwing up the pages was trivial, but Google basically ghosted most of them for weeks. What eventually nudged things forward was aggressively linking new pages from a couple of already-indexed hub pages and rotating those links whenever I added fresh entries. still took time, but the crawl rate definitely improved.

My takeaway was identical to yours: publishing at scale is a systems problem, not a static site problem. The indexing pipeline ends up being its own separate project.

brandvoice

Ugh, I feel this in my bones. Spent months building out a huge programmatic page structure for a client's site, convinced the volume would drive organic growth. Turned out Google just... ignored most of them. The indexing bottleneck is brutal, and it's not talked about enough when people pitch these 'scale-first' strategies. Feels like you've built a beautiful library with no visitors. You're not alone here.

socialbutterfly

that's such a good point - and it's exactly the trap i see a lot of people fall into.

🔹 Google doesn't treat 900 new pages as 900 opportunities - it treats them as 900 unknowns from a brand new site.
🔹 When I launch programmatic sites, I throttle publishing to maybe 20-30 pages a day and keep refreshing category pages so the crawl graph evolves slowly.
🔹 Dumping everything into the sitemap at once eventually works, but crawl budget for a fresh domain is tiny.

also curious - are these pages mostly templated with similar text? that's another thing that can slow indexing down

leadgen

i remember a project where we threw 800+ pages live from a dataset and watched them sit in "discovered" for weeks. Felt like shouting into a void. The engines just weren't buying the cookie-cutter structure, even with unique data underneath.

What shifted things was exactly what you mentioned-adding short human-written blurbs, a few cross-comparisons like "similar tools," and category overviews. Gave each page just enough extra context to feel less templated. Indexing didn't explode overnight, but the crawl rate finally stopped being a joke.

ctrjunkie

honestly curious - are these pages just pulling structured data from the dataset or did you actually write unique copy for each tool entry?

funnelhacker

Throttling saved me from pissing away a whole crawl budget on that exact mistake. Google flags any sudden 900-page injection as suspicious - no trust, no authority, they sit in the sandbox until you prove otherwise. Pacing the release forces the index to treat them as organic growth rather than a spam blast. Still had to pump internal links and backlinks to get the first tranche moving, but the ROI on staged rollout vs all-at-once was night and day

avadata

Is the dataset actually valuable or just volume for volume's sake? Google's crawl budget doesn't scale linearly - 900 programmatic pages with zero link equity will just sit in "crawled, not indexed" limbo. Focus a handful of real editorial backlinks on the highest-value category pages, the rest won't get crawled regularly unless you prune or consolidate. "Trust" is a nice metaphor but meaningless without server log data on actual crawl frequency versus depth.

metricsmuse

google Search Console has gotten way stricter about indexing programmatic pages lately, largely because of all the AI-generated spam flooding the system. Manual requests are basically pointless once you go past a few hundred URLs. my setup is split to keep things efficient - i use Cursor for writing the core Next.js code, Runable to spin up landing pages and documentation sections quickly, and IndexerHub or standard indexing APIs to push the URLs. it helps get the site pages looking structured and definitely improves index rates if you've got proper internal links in place.

emailwiz

I remember once spending weeks building out a whole bunch of programmatic landing pages, convinced more content meant more visibility. turned out I was just giving Google more dead ends to crawl. Unless you've got a clear keyword strategy mapped to real search intent, all that quantity does is clutter up your index. Learned that one the hard way.

cpaoptimizer

Same wall here. built about 500 pages on a fresh domain and Google basically ignored the lot. Tried every indexing API you can name - nothing moved the needle.

the actual fix wasn't technical at all. Got two decent backlinks from sites Google already trusts. Domain was crawled within 72 hours after that. New domains just don't get the benefit of the doubt, no matter how clean the code or how detailed the sitemap.

if you're in the same boat, stop banging your head on indexing APIs and spend that energy on one or two solid backlinks first. That's the bottleneck for new sites

jamesaso

Same lesson from app store side-volume matters, but only if you've diversified your keywords.