Content Discovery Pipelines

I run multi-stage content pipelines across Siteinspire, Popular Archive, and Massive. The stages are the same everywhere: discover, enrich, classify, score. Discovery pulls from RSS feeds, Reddit, Exa, and ScrapingBee depending on the project. Enrichment adds metadata the source doesn't provide: screenshots, word counts, technology detection, author information. Classification runs through an LLM to tag content type, topic, quality signals. Scoring produces a composite number that determines whether the content gets published, queued for review, or dropped.

Deduplication is the unglamorous stage that prevents the whole system from degrading. I hash content with SHA-256 at the enrichment step, checking against every existing hash before proceeding. URLs alone aren't reliable for dedup because the same article appears at different URLs across syndication networks, RSS aggregators, and social sharing. Content hashing catches those duplicates. I also run a lighter fuzzy match on titles to catch near-duplicates: same article republished with a slightly different headline. Between exact hash matching and fuzzy title matching, I filter out roughly 15-20% of incoming content as duplicates across the pipeline.

The source mix matters enormously. RSS gives you consistent, high-quality feeds from publications you trust, but it's a closed loop. You only find what you already know to follow. Reddit surfaces emerging content but with unpredictable quality. Exa provides semantic search across the web, which is good for filling topic gaps. ScrapingBee handles the sites that don't offer feeds or APIs. I weight these sources differently per project. Siteinspire leans heavily on curated RSS feeds and manual submissions. Massive pulls more from Reddit and Exa because it needs breadth across topics. The quality ceiling of your pipeline is determined by your worst source, so I'm aggressive about pruning sources that consistently produce low-scoring content.

The lesson I keep relearning is that quality at scale is a maintenance problem, not a launch problem. The pipeline works well on day one when you've hand-picked your sources and tuned your classification prompts. Six months in, sources drift, new content formats appear that your classifier doesn't handle, and the score distribution skews. I schedule monthly reviews where I sample 50 items from each quality tier and check whether the scores still match my judgment. When they don't, I adjust the prompts, re-weight the scoring dimensions, or drop a source entirely. The pipeline is never done.