Image Vector Search at Materia

Materia's image search lets architects upload a photo of a material — a tile, a fabric swatch, a piece of stone — and find visually similar products across the entire catalogue. The same technology powers the material identification feature, the "similar products" recommendations, and the text-to-image search. It's all the same embedding pipeline with different entry points.

Voyage multimodal-3

The embedding model is Voyage AI's multimodal-3, which produces 1024-dimensional vectors from both images and text. The "multimodal" part is key: an image of a herringbone wood floor and the text "herringbone oak flooring" land near each other in the same vector space. This means the search index built from product images also works for text queries, without maintaining separate indexes.

Every product's main image gets embedded and stored in Typesense with the vector attached. At query time, the search input — whether it's an uploaded image or a typed description — gets embedded with the same model, and Typesense runs KNN (k-nearest-neighbour) search against the indexed vectors.

The image sizing problem

Voyage charges per token, and image tokens scale with pixel count. A 2400px product image costs about $0.0012 to embed. Resizing to 1024px drops the cost to $0.00022 — an 80% reduction — with negligible quality loss for material matching. The perceptual features that matter (texture, pattern, colour distribution) are well-captured at 1024px.

The resize pipeline detects MIME type from magic numbers (not file extension, because URLs lie), then uses Sharp to resize anything over 5MB or 12 megapixels. This catches the oversized product photography that manufacturers love to provide while leaving small images alone.

The PNG transparency trap

This one cost me a day. PNG images with transparent backgrounds, when converted to JPEG for embedding, render the transparent pixels as black. A product photo of a white ceramic tile on a transparent background becomes a white tile on a pitch-black background. The embedding captures "white object on black background" instead of "white ceramic tile," which poisons similarity search — suddenly your white tile matches dark-coloured products because the dominant visual feature is the black surround.

The fix: detect PNG images, composite against a white background before JPEG conversion. Simple once you know about it, invisible until you notice your search results are wrong in ways that seem random but aren't.

Query enhancement

When a text query comes in, the system checks whether it contains known material category terms (film, flooring, carpet, glass, paint, surface, textile, tile, wallcovering, fabric, upholstery). If it doesn't, it appends " materials" to the query before embedding. This is a small hack with a big impact — "dark oak" returns general wood results, but "dark oak materials" returns flooring and surface products specifically. The embedding model's training data responds well to domain-specific context.

Typesense as vector store

I chose Typesense over pure pgvector for the image search index because Typesense gives you vector search and full-text search in the same query. A search for "sustainable wool upholstery" can combine vector similarity on the image embedding with keyword filtering on product attributes, all in one request. pgvector would need a separate full-text search pass.

The index also acts as a vector cache. When a product needs its embedding, the system checks Typesense first. If the vector is already there, no API call needed. If it's missing (new product, or the embedding model was updated), the system generates the embedding on-the-fly and persists it asynchronously (fire-and-forget — the search succeeds immediately, the persistence happens in the background).

Brand diversity and deduplication

Raw vector search returns a lot of near-duplicates. A manufacturer might have 40 variations of the same tile in different colours — they're all visually similar and they'd dominate the results. Two mitigations:

First, deduplication by brand+name combination. If "Marazzi Treverkmore" appears in five colour variations, only one makes it through. Second, brand diversity capping — maximum three products per brand in any result set, selected round-robin so multiple brands get representation even when one brand dominates the vector similarity scores.

Category scoring

Vector similarity alone isn't enough for material matching. A beige limestone and a beige linen might be close in embedding space (similar colour, similar texture at a distance), but they're completely different materials. Category scoring adds a multiplier based on taxonomy alignment — each matching level of the hierarchy (materials → surfaces → stone → limestone) adds to the score. Products in the same L2 or L3 category get a significant boost, which keeps results within the right material family while still allowing the vector similarity to rank within that family.

The compound effect

The same embedding pipeline serves four features: image search (upload a photo, find similar), text search (type a description), material identification (detect and match materials in a room photo), and similar products (find alternatives to a specific product). Building one robust embedding infrastructure and reusing it across features is the kind of compounding that makes the investment in getting the pipeline right worthwhile. Every improvement to image processing or embedding quality improves all four features simultaneously.