adult site crawled but not indexed - trust or tech?

paperclick

We launched a Danish sex toy price comparison site - think PriceRunner for adult products. Same pSEO setup that got another site 1M impressions and 28k indexed pages in three months. here? 837 impressions, 3 clicks (probably mine), and exactly one indexed page. Google is crawling nearly 2k pages per GSC but refusing to index them.

A colleague looked at the setup and spotted a few killers:

First, we were blocking /go/ in robots.txt while also using rel="nofollow sponsored" on those affiliate links. Telling Google "come see how great my site is, but I won't show you where users go" is a trust signal problem - especially in an adult niche.

Second, way too many pages for a new domain. Sitemaps showed 10k+ product URLs, 6k+ comparisons, 2k+ cross/facet landing pages. That's a massive crawl budget ask for a site with no authority.

Third, conflicting canonicals on some comparison pages - self-canonical plus another pointing to /sammenlign/. Plus sitemap URLs that are actually noindexed. That's a technical mess.

the real lesson: unique content isn't enough anymore. lLMs killed that. Google wants to know if a page genuinely helps someone choose - original testing, comparison logic, editorial judgment, price history. Most of our pages were just unique strings. Not valuable.

My fix list: remove the /go/ block, clean up canonical conflicts, prioritise category pages and guides over product pages, add meta name="rating" content="adult", and stop treating pagination like a problem - let Google index paginated URLs if it wants.

Oh, and the backlink profile is questionable. Some came from link platforms. If Google hasn't already caught the pattern, it will.

bottom line: it's a trust issue first, technical second. Google sees a new adult affiliate site with blocked outbound links, too many similar URLs, and not enough proven value. Fix the signals, then let the content earn indexation

ppcnerd

I had a look at the site and also ran some sampling with Screaming Frog to get a sense of the structure and implementation.

The first that catches my attention is the /go/ block in robots.txt. Your affiliate links are already marked with rel="nofollow sponsored noopener", which is the right direction for paid/affiliate links - while you could remove nofollow as it as originally created for links in blog comments and sponsored is the correct definition here.

You are basically telling Google; Hey, come and see how great my site is, so you can rank it, but I am not going to show you where I send the users, because I decided to both cloak and block the URLs. This doesn't really rhyme with "helpful content".

From the sitemaps I saw roughly:

- 10k+ product URLs

- 6k+ comparison URLs

- 2k+ cross/facet-style landing pages

- hundreds of brand/category/facet URLs

That is a lot for a new affiliate/comparison site, especially in an adult niche.

A few things I would fix first:

Remove the disallow of /go in robots.txt
Some sitemap URLs are noindexed. Only canonical, indexable URLs should be in XML sitemaps.
Some comparison pages have conflicting canonicals: a self-canonical plus another canonical to /sammenlign/. That is a serious signal conflict.
Prioritize category pages, best guides, and comparison pages that genuinely help users choose. Make sure you have a great tracking plan and detailed custom event tracking implemented, so you can actually tell how the users engage with the site. This is the signal Google is looking after today!
Add explicit/adult metadata where appropriate, e.g. <meta name="rating" content="adult">.
I'd be a bit skeptical of the backlink profile. Sooner or later (if not already) Google will probably catch up to the pattern.

On the content point: "non-spun" and "unique" content is not enough anymore. That was a 2018 way of thinking. Google can crawl a page, see that the words are unique, and still decide it does not add enough value to index. Unique content is definitely not a commodity anymore after LLMs entered the scene. Content quality is. And entity-coverage is what they are looking for beyond user signals.

The real question is not "is this text unique?" but:

- Is this page more useful than the shop/product pages it sends users to?

- Does it help the user choose better, faster, or safer?

- Is there original testing, real comparison logic, editorial judgment, pros/cons, price history, availability, merchant trust data, or something users cannot get elsewhere?

- Would this page deserve to rank if affiliate revenue was removed from the equation?

Right now my guess is: Google sees a new adult affiliate/comparison site with blocked outbound links, a huge number of similar commercial URLs, mixed canonical/noindex signals, and not enough proven unique value yet. So it crawls, but chooses not to index most of it.

neonlights

Zero indexation = zero activation. here's why your crawl's worthless. missing Schema kills any chance of showing as a comparison site - no structured data, no SERP real estate. hotlinking screams "don't list me", that's a direct authority penalty. cloaked links? That's a rapid de-indexation path. You've got four pages indexed, not one, but with no links and zero domain authority even pSEO can't fix that. Fix Schema, cut hotlinking, ditch cloaks. basics aren't optional in this game

sleepermode

Your .htaccess file might be issuing a Disallow directive via robots.txt if you're generating it dynamically, while the actual page-level <meta name="robots" content="index"> tags say the opposite. that conflict is a classic trap-crawlers see the Disallow and back off, but your pages still signal they're indexable.

if you'd previously proxied through Cloudflare and then moved to standard hosting, the old cache could have a stale robots.txt that wasn't regenerated. google Search Console's Index Coverage report will surface that discrepancy under "Submitted URL not selected" or "Crawled - currently not indexed". Also pull the live robots.txt check in GSC to confirm what Google actually sees.

For a price comparison site, you're likely dealing with faceted navigation or parameter-heavy URLs-those can also get stuck if your .htaccess is blocking the canonical versions. Worth auditing your crawl logs alongside the GSC data

dusting

From what I've seen, Google crawling but not indexing usually points to a site authority bottleneck. Content quality alone doesn't determine indexation-I had a 120-word page indexed within an hour. The duplicate content check is based purely on the URL and the title tag. If authority is thin, even solid pages sit in a crawl queue.

paperclick

Absolute gold, mate. Thanks for sharing that.

I'd be careful with blanket no-indexing all product pages. Instead, I'd prune the ones with zero clicks and impressions in Search Console - no point keeping 5,000 near-identical product pages alive if they're just diluting your crawl budget. Focus your index on the 100-200 that actually get organic traction, then build out the informational content (guides, comparisons) as your authority hubs. That's worked well for me in similar markets.

On the backlinks: don't manually disavow them unless you've seen a manual action or a ton of toxic links. Google's pretty good at ignoring low-quality link building noise these days. Spend that energy on building one or two genuinely relevant links instead.

As for the unique angle - quizzes and price history are good, but the real test is whether they solve the searcher's intent better than anyone else. If you're just layering gimmicks on top of a generic comparison template, Google will still see it as thin. I'd run a few A/B tests on the top-performing pages to see which format actually keeps people on page and drives conversions. Numbers don't lie.

ppcnerd

Glad you found that useful.

There's a principle I've noticed across both paid and organic channels: when you throw every available signal at a problem at once, you lose the ability to understand what actually moved the needle. It's like trying to tune an engine by replacing every part simultaneously - you might end up with a working car, but you won't know which fix mattered. I see that pattern in your setup. cloaked affiliate URLs, robots.txt blocks, rel=nofollow, rel=sponsored - each one alone is a reasonable tactic, but together they create a web of contradictions that confuses both crawlers and your own troubleshooting.

Before you layer on another solution, stop and ask: what specific problem am I solving, and what is the simplest way to verify that solution worked? Otherwise you're left with too many moving parts and no clear feedback loop. Sometimes the quietest signal is the most effective.

I'd start by removing the /go/ blocking entirely. Let Google crawl those URLs naturally, then check your server logs - a simple log analyser (there are free tools or you can ask a LLM to build one for your specific needs) will show you if the crawler is actually hitting those paths. Give it a week or two of data. if the indexed page count doesn't budge, then the blocking wasn't the bottleneck.

from there, focus on the evergreen content that outlives any single product: category pages, guides, quizzes. look at each page from the outside - ask what questions a visitor has that aren't answered clearly. lLMs are brilliant for this kind of content audit if you spend time getting the prompts right.

A few additional things I noticed on your pages:

Pagination handling is over-engineered. You block crawl of paginated URLs, set canonicals to the main category, and slap noindex on top. That's three contradictory instructions. If you block the page, Google can't see your canonical or noindex tag anyway. Let paginated URLs be crawled and indexed - Google will decide which fits a query. If ?side=3 starts ranking when it shouldn't, the fix is to de-optimise those versions (e.g. remove <h1> but keep the visual styling with a <span>). Google's own docs on e‑commerce pagination cover this.
I spotted duplicated <link rel="dns-prefetch"> entries on at least the category pages. Not a critical bug, but easy to clean up.
The FAQ answers in your categories feel a bit thin. they read as if written to check a box rather than serve a user. Take the time to prompt a LLM to generate both questions and genuinely helpful answers - you can create something that outperforms what's already ranking. It's a small investment that pays off across dozens of pages.

paperclick

That's genuinely the most actionable feedback I've had in months.

Quick backstory: I've run SEO on other sites using Yoast and Rank Math, but on this domain I went all-in on Claude Code as the SEO engine. The technical foundation was solid, but when GSC numbers flatlined I started prompting more defensively - "make sure X is not indexed", "block /go/ just in case", "noindex paginated pages". Claude obediently stacked every available signal on top of each other. Your "less is more" hit where it needed to.

Acting on it in iterations so I can see what actually moves the needle. Iteration 1 done: /go/ unblocked, duplicated DNS prefetch hints removed, redundant Sitemap line in robots.txt fixed, adult rating meta added. Iteration 2 planned: drop the pagination hammer, fix the dual canonical on /sammenlign/, rewrite the auto FAQ.

One question: apart from backlinks, any tricks to show Google authority on a fresh domain?

Seriously appreciate the time you've put into this.

ppcnerd

Great! Happy to hear that you find it useful. I've been doing SEO for 20+ years - so back to when SEO analysis was mainly done by hand, but moved into being more of a Swizz knife for web today, working with fever and larger clients and more on strategy, meaning I don't get to see edge cases very often, so to stay sharp I turn to threads like this from time to time.

Ha! I am so biased when it comes to Claude that I wouldn't necessarily listen to my opinion on it. While I do find it creative and good at understanding intent in general and reading between the lines, I also find it to be too creative for some tasks. With creativity comes randomness, and for some tasks you want something that is more deterministic and rigid in it's approach. I used Claude Code day and night from around April 2025 but had enough sometime around September 2025. They were definitely switching the dials for performance in the background while not being transparent about it, so I switched to Codex which might not be too good at UI tasks, but nails everything else in a more reliable way.

Regarding GSC number stagnating; that is quite normal on a new domain. Google will give you a grace period where they boost your rankings to give you traffic enough to test your site against the competitors. Once that period is over you will often see a decline in traffic. This doesn't necessarily mean that you did anything wrong - just that you are back to basic organic growth. Keep on pushing the quality of your site, but don't act on it as if it's a penalty. It you do it right, the numbers will pick up again.

On authority for a new domain; the right links will do the trick, but so will true engagement. People staying on and coming back to your site because they see it as a valuable tool, searching for it on Google, etc. If you want to dive into it there's a lot of gold from the DOJ vs Google trial here: https://www.hobo-web.co.uk/google-vs-doj/

You know, Google preached for many years, that site owners should optimize for humans, and the rankings would follow. While that definitely wasn't the full story 10 years ago, we are actually getting quite close to that reality now. It is still just math, but a different kind of math.

paperclick

Absolutely! And it’s interesting that you’re here specifically for edge cases. If you ever need another Danish edge case, just let me know. My other pSEO project is full of edges.

Interesting with 20 years in the SEO world. I’ve only worked with SEO for a little over a year at an agency, and that was mainly onsite SEO. The knowledge you seem to have around technical SEO feels very high level and probably something you work with daily, so not to sound too fanboy-ish, but is there anywhere else to follow you besides Reddit? Like a blog, posts, or anything similar?

Hehe. I was actually the opposite. I switched from GPT to Claude, though back in November/December 2025. I felt GPT struggled with handling too many lines of code at once without hallucinating too much. But I can definitely sense here and on X that more people are praising Codex lately. I should probably test it out again.

And thanks for the link. That’s definitely going to be tonight’s reading material. Stuff like that is genuinely fascinating.

Regarding writing and optimizing for people instead of Google, that’s something I’ve started experimenting with more and more. I’ve had a tendency to write way too much for keywords and add extra sections just to hit a word count target. But I also think that the more information fatigued we become because of AI, the less people want to read long-form content. For example, I’ve started testing a kind of TL;DR section at the beginning of articles to see if it sends a signal that the content is made for the reader and not just for SEO. I’m really looking forward to seeing more data from GSC on that part 😅