# Pinterest-Style Search & Pin Creation — Design Document

Scope: Pin creation (image + description) and search (text query, suggestions, masonry grid results)
Out of scope: Video uploads (mp4), canvas-based pin editor, boards, social graph, recommendations

# 1. System Overview

# Scale

Anchor: 10M pins, 50K daily active users.

At this scale: Elasticsearch handles full-text search with sub-50ms p99 query latency via Redis caching and a lean index (description + suggest only). Image storage and CDN egress dominate cost — WebP variants, immutable UUID URLs, and aggressive CDN caching keep egress near-zero at steady state. CDC (not dual-write) keeps the search index consistent without unbounded polling cost as pin volume grows.

# Stack

• Frontend: Next.js (React) — App Router with React Suspense streaming
• Backend: Node.js API
• Primary DB: PostgreSQL
• Search index: Elasticsearch
• Object storage: S3 (pin images, presigned upload)
• CDN: CloudFront / Cloudflare (SSR HTML + image variants)
• Cache: Redis (60s TTL on search/suggestions)
• Image processing: Sharp worker (WebP variants, dominant color extraction)
• Queues: SQS — processing queue (S3 upload → Sharp) and index queue (CDC → Elasticsearch)
• CDC: Debezium (Postgres WAL → index queue)

# Surfaces

Two user-facing surfaces:

┌─────────────────────────────────────────────────────────────────┐
│                          Browser                                 │
│   Next.js App (React Suspense streaming / CSR per route)        │
└─────────┬──────────────────────────┬────────────────────────────┘
          │ REST                     │ presigned URL upload
          ▼                          ▼
┌──────────────────┐        ┌─────────────────┐
│   Node.js API    │        │   S3 (images)   │◄── CDN (CloudFront)
└──────┬───────────┘        └────────┬────────┘
       │ writes                      │ ObjectCreated event
       ▼                             ▼
┌──────────────┐           ┌──────────────────┐
│  PostgreSQL  │           │  SQS (proc queue) │
│  (source of  │◄──────────│                  │
│    truth)    │  update   └──────┬───────────┘
└──────┬───────┘                  │
       │ WAL                      ▼
       ▼                  ┌──────────────┐
┌──────────────┐          │ Sharp worker │ (generates variants,
│  CDC process │          │              │  extracts dominant color,
│  (Debezium)  │          └──────┬───────┘  updates Postgres)
└──────┬───────┘                 │
       │                         │
       ▼                         │
┌──────────────┐                 │
│ Index queue  │                 │
│ (SQS/Kafka)  │                 │
└──────┬───────┘                 │
       ▼                         │
┌──────────────────┐             │
│ Indexing consumer│             │
└──────┬───────────┘             │
       ▼                         │
┌──────────────────┐    ┌────────▼──────────┐
│  Elasticsearch   │    │  Redis (cache)    │
│  (search index)  │    │  60s TTL          │
└──────────────────┘    └───────────────────┘

# 2. Data Model

# PostgreSQL — pins table

CREATE TABLE pins (
  id             UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  description    TEXT NOT NULL,
  s3_key         TEXT NOT NULL,           -- pins/{id}/original.{ext}
  width          INTEGER NOT NULL,        -- original image dimensions
  height         INTEGER NOT NULL,        -- used for masonry pre-calc
  dominant_color CHAR(7) NOT NULL,        -- e.g. "#a3b4c5"
  status         TEXT NOT NULL            -- uploading | processing | ready
                 DEFAULT 'uploading',
  created_at     TIMESTAMPTZ NOT NULL DEFAULT now()
);

CREATE INDEX pins_status_created_at ON pins (status, created_at DESC);

Image URLs are derived, never stored — constructed from s3_key:

https://cdn.example.com/pins/{id}/236w.webp   ← search grid (srcset small)
https://cdn.example.com/pins/{id}/474w.webp   ← search grid (srcset large / retina)
https://cdn.example.com/pins/{id}/736w.webp   ← pin detail page

# Elasticsearch — pins index

Only searchable fields are indexed. Full pin data stays in Postgres.

{
  "mappings": {
    "properties": {
      "id": { "type": "keyword" },
      "description": { "type": "text", "analyzer": "english" },
      "created_at": { "type": "date" },
      "suggest": {
        "type": "completion",
        "analyzer": "simple"
      }
    }
  }
}

The suggest field is populated from tokenised description text — powers the suggestion API via ES completion suggester.

# 3. API Design (REST)

# Pin Creation

# POST /api/pins/upload-url

Request a presigned S3 URL. Creates the pin row in Postgres immediately with status: uploading.

Request:  { content_type: "image/jpeg", file_size: 8388608 }
Response: { pin_id: "uuid", upload_url: "https://s3.../...", expires_in: 300 }

Validation: content_type must be image/jpeg or image/png. file_size must be ≤ 20MB. Rejected immediately — no S3 key issued.

# POST /api/pins

Submit pin metadata after upload completes.

Request:  { pin_id: "uuid", description: "..." }
Response: { pin_id: "uuid", status: "processing" }

S3 ObjectCreated event (not this endpoint) triggers the Sharp processing job.

# GET /api/pins/:id

Returns a single pin for the detail page. Reads from Postgres.

# Search

# GET /api/search?q={query}&cursor={cursor}

Returns the first (or next) page of results from Elasticsearch.

{
  "pins": [
    {
      "id": "uuid",
      "description": "Best Things to Do in Flam Norway",
      "width": 1024,
      "height": 1536,
      "dominant_color": "#4a7c59",
      "images": {
        "236w": "https://cdn.example.com/pins/uuid/236w.webp",
        "474w": "https://cdn.example.com/pins/uuid/474w.webp"
      }
    }
  ],
  "next_cursor": "opaque_search_after_token"
}

• Ranking: BM25 relevance combined with function_score exponential decay on created_at (scale: 30 days).
• Pagination: search_after — no offset. Cursor is the sort values of the last document, base64-encoded.
• Caching: Redis checks before hitting ES. Key: search:{sha256(q)}:{cursor}. TTL: 60s.

# GET /api/suggestions?q={query}

Returns up to 8 suggestion strings from ES completion suggester.

{
  "suggestions": [
    "keyboards mechanical",
    "keyboards gaming",
    "keyboards aesthetic"
  ]
}

Cached in Redis. Key: suggest:{sha256(q)}. TTL: 60s.

# 4. Pin Creation Flow

Browser                  API                    S3              SQS           Sharp Worker       Postgres
  │                       │                      │               │                │                 │
  │ POST /upload-url       │                      │               │                │                 │
  │──────────────────────►│                      │               │                │                 │
  │                       │ INSERT pin            │               │                │                 │
  │                       │ status=uploading      │               │                │        ─────────►│
  │                       │ generate presigned URL│               │                │                 │
  │◄──────────────────────│                      │               │                │                 │
  │                       │                      │               │                │                 │
  │ PUT image (binary)    │                      │               │                │                 │
  │──────────────────────────────────────────────►               │                │                 │
  │◄─────────────────────────────────────────────               │                │                 │
  │                       │                 ObjectCreated event  │                │                 │
  │                       │                      │──────────────►│                │                 │
  │                       │                      │               │ enqueue job    │                 │
  │                       │                      │               │───────────────►│                 │
  │ POST /pins (metadata) │                      │               │                │                 │
  │──────────────────────►│                      │               │                │                 │
  │◄──────────────────────│                      │               │                │                 │
  │ status: processing    │                      │               │                │                 │
  │                       │                      │               │  run Sharp     │                 │
  │                       │                      │               │  236w/474w/736w│                 │
  │                       │                      │◄──────────────────────────────│                 │
  │                       │                      │  store variants               │                 │
  │                       │                      │               │  UPDATE pin    │                 │
  │                       │                      │               │  status=ready  │        ─────────►│
  │                       │                      │               │  +dominant_color│                │

After status=ready is written, the CDC process picks up the WAL event and routes it to the indexing consumer, which writes to Elasticsearch. The pin appears in search within seconds.

# 5. Search Flow

Browser (Next.js)              API + Redis              Elasticsearch
  │                                 │                        │
  │  GET /search?q=keyboards        │                        │
  │  (initial page load — SSR)      │                        │
  │────────────────────────────────►│                        │
  │                                 │ Redis HIT → return     │
  │◄────────────────────────────────│                        │
  │  HTML shell streamed first      │                        │
  │  Pin data streams inline        │  Redis MISS            │
  │  (React Suspense boundary)      │────────────────────────►│
  │                                 │◄────────────────────────│
  │                                 │ write to Redis         │
  │◄────────────────────────────────│                        │
  │  Hydration: no second fetch     │                        │
  │                                 │                        │
  │  [user scrolls to bottom]       │                        │
  │                                 │                        │
  │  GET /api/search?q=keyboards    │                        │
  │  &cursor=opaque_token  (CSR)    │                        │
  │────────────────────────────────►│                        │
  │◄────────────────────────────────│                        │
  │  Append new pins to grid        │                        │

# 6. Frontend Architecture

# Routing & Rendering

The Next.js App Router loading.tsx file provides the immediate shell. A <Suspense> boundary wraps the pin grid — the server streams pin data into it as the ES query resolves.

# Search suggestions

# Pin creation

# Masonry Grid

┌──────────┐  ┌──────────┐  ┌──────────┐
│          │  │          │  │          │
│  Pin A   │  │  Pin B   │  │  Pin C   │
│ h=320px  │  │ h=480px  │  │ h=260px  │
│          │  │          │  │          │
└──────────┘  │          │  └──────────┘
┌──────────┐  │          │  ┌──────────┐
│          │  └──────────┘  │          │
│  Pin D   │  ┌──────────┐  │  Pin E   │
│ h=200px  │  │          │  │ h=400px  │
│          │  │  Pin F   │  │          │
└──────────┘  │ h=360px  │  │          │
              │          │  └──────────┘
              └──────────┘

Layout algorithm:

1. On mount, read container width → calculate column count (e.g. 2 on mobile, 3 on tablet, 4 on desktop).
2. Track columnHeights[]. For each pin, place it in the shortest column. Position: { top: columnHeights[col], left: col * (colWidth + gap) }.
3. Pin height is known from Postgres (stored at upload time) — calculated as (stored_height / stored_width) * colWidth. Zero layout shift; no waiting for images to load.
4. Container height = Math.max(...columnHeights).

Loading placeholder: Each pin slot renders with background-color: dominant_color immediately. The <img> crossfades in on load. On onerror: wait 2s, retry once. If retry fails, show dominant color + broken-image icon. Slot never collapses (CLS violation).

Virtualization: Only pins whose calculated top is within [scrollY - overscan, scrollY + viewportHeight + overscan] are rendered. Overscan = 1 viewport height. Unmounted pins leave their slot height intact (a div with the pre-calculated dimensions), so scroll position is stable.

Image markup (grid card):

<img
  src="https://cdn.example.com/pins/uuid/474w.webp"
  srcset="
    https://cdn.example.com/pins/uuid/236w.webp 236w,
    https://cdn.example.com/pins/uuid/474w.webp 474w
  "
  sizes="(max-width: 600px) 50vw, (max-width: 900px) 33vw, 25vw"
  fetchpriority="high"
  ←
  first
  visible
  row
  only
  loading="lazy"
  ←
  all
  others
  alt="Best Things to Do in Flam Norway"
  width="474"
  height="711"
/>

Paint scheduling:

# Search Bar & Suggestions

• Input is debounced at 200ms before firing GET /api/suggestions.
• Each new keystroke calls AbortController.abort() on the previous in-flight request before creating a new one. Prevents stale suggestions from a slow earlier query rendering after a faster later one.
• Suggestions rendered as a dropdown role="listbox" / role="option" list. Keyboard: ↑/↓ navigate, Enter submits, Escape closes.

# Accessibility

# 7. Image Processing Pipeline

At upload time, the Sharp worker performs all transforms in a single pass:

1. Validate — reject if not JPEG/PNG, abort if corrupt.
2. Extract metadata — width, height of original.
3. Extract dominant color — quantize to 1 color, store as hex.
4. Generate WebP variants — 236w, 474w, 736w, quality 80.
5. Write variants to S3 — pins/{uuid}/236w.webp etc.
6. Delete original from S3 — cost control.
7. UPDATE pin — set status=ready, dominant_color, width, height.

CDN cache headers on all variant objects:

Cache-Control: public, max-age=31536000, immutable

URLs are UUID-keyed and content never changes — safe for permanent caching.

# 8. Elasticsearch Sync (CDC)

Postgres WAL
    │
    ▼
CDC process (Debezium)
    │  publishes row-change events
    ▼
SQS / Kafka topic: pin-changes
    │
    ▼
Indexing consumer
    │  filters: only process events where new status = "ready"
    │  (ignores uploading → processing transitions)
    ▼
Elasticsearch upsert
    { id, description, created_at, suggest: [tokenised description terms] }

Failure mode: If Elasticsearch is down, events queue in SQS. The consumer retries with exponential backoff. When ES recovers, events replay in order. The Postgres row is the source of truth — a full re-index is always possible by replaying from created_at order.

# 9. Caching

Browser request: GET /search?q=keyboards
       │
       ▼
CDN (first page, hot queries)
  HIT → return cached SSR HTML (sub-10ms)
  MISS ↓
       │
       ▼
Node.js API
       │
       ▼
Redis  HIT → return JSON (< 5ms)
  MISS ↓
       │
       ▼
Elasticsearch (~20–50ms)
       │
       ▼
Write to Redis (TTL: 60s)
Return response

# 10. Observability

Signals tracked, frontend-first:

1. Core Web Vitals (per page, real user monitoring)

   • LCP on /search — time for first image row to paint. Target: < 2.5s.
   • CLS on /search — masonry slot stability. Target: < 0.1. Alert if any slot collapses on image load or error.
   • INP on /search — scroll interaction responsiveness. Target: < 200ms.

2. Search latency p50/p95/p99 — measured at the API layer (excludes CDN hits). A p99 spike → check ES cluster health first, then Redis hit rate.

3. CDC lag — time delta between pins.created_at and the ES document's index timestamp. Growing lag → indexing consumer falling behind. Queue depth is the leading indicator; trigger alert at > 5 minutes lag.

4. Pin creation funnel — four-stage success rate tracked per request:

   • Presigned URL issued
   • S3 upload confirmed (ObjectCreated received)
   • Sharp processing complete
   • status=ready written

   A drop at any specific stage isolates the failure (S3 issue vs Sharp crash vs Postgres write failure).

# 11. CDN Strategy & Availability

# Multi-CDN fallback

CDN availability is business-critical for this system — images and SSR HTML both route through it. Cloudflare (June 2022) and Fastly (June 2021) each caused widespread outages that took down significant portions of the internet. A single-CDN dependency is an unacceptable SPOF at production scale.

Approach: active/passive multi-CDN with usage-based pricing

Configure two CDN distributions (e.g. Cloudflare as primary, CloudFront as secondary) both pointing at the same origin. Traffic routing via DNS failover (Route 53 health checks, or Cloudflare Load Balancer with health monitors). The secondary CDN is cold — it serves no traffic during normal operation. Because both CDNs use usage-based pricing (you pay for bytes served and requests made, not reserved capacity), the passive distribution costs effectively nothing when idle. On primary CDN failure, DNS TTL-based failover routes traffic to the secondary within 30–60 seconds.

                        ┌─────────────────────────┐
                        │   DNS Health Routing     │
                        │  (Route53 / CF LB)       │
                        └──────────┬───────────────┘
                    primary  ──────┘    └────── fallback (cold)
                        │                          │
               ┌────────▼───────┐      ┌───────────▼──────┐
               │  Cloudflare    │      │   CloudFront /    │
               │  (primary)     │      │   GCP Cloud CDN   │
               └────────┬───────┘      └───────────────────┘
                        │ (both origins point to same object storage)
                        ▼
               ┌─────────────────┐
               │  R2 / S3 / GCS  │
               └─────────────────┘

Cost impact: Near-zero. The secondary CDN serves no traffic in steady state — no bytes billed, no requests billed. The only cost is the health check pings (negligible). This is the reason usage-based CDN pricing makes multi-CDN practical: reserved-capacity CDNs would charge for idle standby.

Cloud-specific CDN pairings:

All three work with the same URL pattern — switching CDN is a DNS change, not an application change.

# 12. Cost Controls

# 13. Trade-offs & Key Decisions

See docs/adr/ for full context. Summary:

# 14. Future Extensions

• Canvas-based pin editor — Fabric.js/Konva.js layer system. Output: composited flat image exported to S3. Replaces the simple file upload form.
• Video pins — mp4 up to 200MB. Requires transcoding pipeline (separate from Sharp), HLS streaming, different CDN configuration.
• Search ranking v2 — fold save_count engagement signal into function_score once event tracking is in place.
• Personalised suggestions — weight ES completions by user's past search terms (requires session history store).
• Virtual scroll with recycled DOM nodes — replace current "render/unmount near viewport" approach with a fixed pool of recycled card elements for lower GC pressure at extreme scroll depth.