Material Identification with Vision Models

The material identification feature in Materia lets architects photograph a room and identify every material in it — the flooring, the wall treatment, the upholstery, the countertop. Then it finds matching products in the catalogue for each one. The pipeline is three stages: detect what materials are in the image, segment each one, and search for similar products.

Detection with Gemini Flash

The first stage uses Gemini Flash 3 as a multimodal vision model. It receives the image and a list of the 236 material categories from our three-level taxonomy (ltree format: materials.textiles.upholstery, materials.flooring.lvt, and so on). The model returns up to six materials, each with a natural language label ("beige linen upholstery"), a bounding box with normalised coordinates, and a category key matched against the database.

Category matching has a fallback chain: exact match, then case-insensitive, then partial string match. This matters because the model occasionally returns category names that are close but not exact — "interior_floor_tile" vs "interior-floor-tile" — and failing to match would mean the user sees "unknown material" for something the model correctly identified.

There's a classification step before detection that determines the image type: swatch (single material filling the frame), installation (a room with multiple surfaces), or flatplan (material samples arranged like a mood board). This classification changes the downstream behaviour — a swatch goes straight to vector search, while an installation needs per-material segmentation.

The classification uses a two-question prompt technique. Instead of asking "what type is this?" directly, I first ask "are there multiple separate material samples visible?" as a boolean, then use that answer to classify. Forcing the model through explicit logical steps produces more reliable classifications than a single open-ended question.

From detection to matching

Each detected material gets its own pipeline. The bounding box crops the original image to isolate the material, then a Voyage AI embedding (multimodal-3, 1024 dimensions) converts the crop into a vector. That vector goes into Typesense for approximate nearest-neighbour search against the product catalogue.

The matching algorithm is more nuanced than raw vector distance. Results get scored by category match (how many levels of the taxonomy hierarchy align), filtered for brand diversity (max three products per brand, round-robin selection), and optionally reordered by colour similarity using OKLab distance on the dominant colours. Category matching is crucial — a beige textile should match other beige textiles, not beige stone, even if the raw vectors are closer.

A practical optimisation: images over 5MB or 12 megapixels get resized to 1024px width before embedding. This reduces Voyage API costs by about 80% with negligible quality loss for material matching. PNG transparency gets flattened against a white background before JPEG conversion — without this, transparent areas compress to black artifacts that poison the embedding.

The attribute system

Each product in the catalogue has up to 165 JSONB attributes split between codeset attributes (90 categorical values like lamping, pattern, construction, abrasion rating) and non-codeset attributes (75 text/numeric/boolean values like designer name, warranty, width, thickness, composition). The schema is the result of mapping an industrial materials data warehouse into a format that supports both filtering and semantic search.

Two derived scores — sustainability percentile and durability percentile — rank each product relative to its category. A wallcovering with a recycled content of 25% might be in the 80th percentile for wallcoverings but the 30th percentile for textiles. Category-relative scoring makes comparisons meaningful.

State machine for multi-material mode

The UX challenge is that a single image can contain six materials, each at a different stage of processing. The match store is a state machine with two paths: swatch mode (idle, detecting, swatch-ready, matching, complete) and scene mode (idle, detecting, scene-ready, extracting-all, multi-ready, multi-complete). Each individual material within a scene tracks its own state: pending, segmenting, cropping, searching, complete, or error.

This granularity matters for the interface. The user sees each material progressing independently — one might already have search results while another is still being segmented. Without per-material state tracking, you'd either have to wait for everything to finish (slow) or show nothing until everything is done (frustrating).

Product extraction from the web

There's a parallel pipeline for extracting product data from manufacturer websites. Gemini Flash 2.5 processes cleaned HTML to extract name, brand, description, images, and a category hint. The HTML cleaning is aggressive: remove scripts, styles, SVGs, templates, iframes, comments, base64 data URIs, event handlers, framework attributes, hidden elements. The goal is to get the page down to about 30K characters (~7.5K tokens) while preserving product information.

Image extraction is especially tricky. URLs get pulled from raw HTML before cleaning (to capture aria-hidden content), then filtered to remove favicons, icons, logos, tracking pixels, and social media images. CDN images get priority, and URLs containing quality hints like "2500px" or "full" get ranked higher.

Fail fast

The entire pipeline follows a fail-fast philosophy. Vision model detection times out at 60 seconds. Classification times out at 10 seconds. There are no fallbacks and no silent degradation. If the model fails, the user sees an error immediately rather than getting bad results minutes later. This was a deliberate choice — in a professional tool, wrong results are worse than no results, and "I couldn't identify this material" is more honest than a confident but incorrect match.