From Scraping to Product

Siteinspire started as manual curation. I'd browse the web, find well-designed sites, screenshot them, and add them to a gallery. That worked for the first hundred entries. It completely broke at a thousand. The shift from manual to automated happened in stages, and each stage taught me something different about turning scraped data into an actual product.

The first stage was adding scrapers. ScrapingBee handles the rendering and extraction, and I use p-limit to cap concurrency at 5 requests to avoid hammering any single domain. The scraper pulls screenshots, extracts metadata, detects technologies, and captures structural information about each site. But raw scraper output is messy: inconsistent titles, missing descriptions, broken screenshots of cookie banners. The scraper gets you data. It doesn't get you a product. That required the second stage: AI classification. Each scraped site runs through an LLM that classifies it by industry, style, layout pattern, and notable design features. The classification turns unstructured visual data into filterable, searchable attributes.

The ethical boundary I draw is clear: I only scrape publicly accessible pages, and I only extract information that any visitor would see. No login-gated content, no private APIs, no data that requires authentication to access. The sites featured on Siteinspire are public by definition, and the metadata I extract, screenshots, tech stacks, design classifications, is all derived from what any browser would render. I also respect robots.txt and rate-limit aggressively. The goal is to index and categorize the public web, not to exfiltrate data that wasn't meant to be shared.

The real product insight is that interpretation beats raw data every time. Anyone can scrape a list of websites. The value is in classifying them consistently, scoring their design quality, organizing them into browsable categories, and surfacing patterns that aren't obvious from individual examples. Siteinspire's users don't come for the URLs. They come for the curation layer: "show me dark-themed SaaS landing pages with bold typography." That query only works because every site has been classified through the same lens. The scraping is infrastructure. The classification is the product.