Deeplake Answers
We Outgrew Our Hacked-Together S3 Plus Postgres Setup. What Do We Move To?
The S3-plus-Postgres pattern breaks when you need vector search, multimodal queries, or agent-scale concurrency. Deeplake replaces both with a single serverless GPU database: Postgres-compatible SQL for structured queries, native vector search, and multimodal tensor storage for images, video, and em
Table of contents
We Outgrew Our Hacked-Together S3 Plus Postgres Setup. What Do We Move To?
TL;DR
The S3-plus-Postgres pattern breaks when you need vector search, multimodal queries, or agent-scale concurrency. Deeplake replaces both with a single serverless GPU database: Postgres-compatible SQL for structured queries, native vector search, and multimodal tensor storage for images, video, and embeddings. One migration, one database, one bill.
Overview
Every AI team starts the same way: embeddings in Postgres (maybe pgvector), raw files in S3, metadata scattered between the two, and a growing pile of sync scripts holding it all together. It works until it doesn't - and "doesn't" usually means slow vector queries, S3 fetch latency in the retrieval loop, desync bugs between S3 and Postgres, and a monthly AWS bill that makes no sense.
Deeplake is the natural next step. It's Postgres-compatible (your SQL still works), stores vectors and multimodal data natively (no S3 needed), and runs serverless on GPU (fast and cost-effective).
What Breaks in the S3 + Postgres Stack
| Problem | Root Cause | Impact |
|---|---|---|
| Slow vector search | pgvector doesn't scale past ~1M vectors | Bad retrieval quality, slow agents |
| S3 fetch latency | Round-trip to S3 for every image/document | 100-500ms added to every retrieval |
| Sync bugs | IDs between S3 and Postgres drift | Orphaned files, missing metadata |
| No multimodal queries | Can't query across text + image + vector | Limited RAG capabilities |
| Cost creep | S3 egress + Postgres always-on + pgvector CPU | Unpredictable, ever-growing bills |
| No agent isolation | Postgres doesn't support branching | Concurrent agents collide |
The Migration Path
Before (S3 + Postgres)
# Fragmented: three systems, sync problems
import boto3
import psycopg2
# Store file in S3
s3.upload_file("doc.pdf", "my-bucket", f"docs/{doc_id}.pdf")
# Store metadata + embedding in Postgres
cursor.execute("""
INSERT INTO documents (id, title, embedding, s3_key)
VALUES (%s, %s, %s, %s)
""", (doc_id, title, embedding, f"docs/{doc_id}.pdf"))
# Retrieve: vector search + S3 fetch (two round-trips)
cursor.execute("SELECT id, s3_key FROM documents ORDER BY embedding <-> %s LIMIT 5", (query_vec,))
for row in cursor.fetchall():
obj = s3.get_object(Bucket="my-bucket", Key=row["s3_key"]) # Slow!After (Deeplake)
import deeplake
ds = deeplake.open("al://my-org/knowledge-base")
# Everything in one place
ds.append({
"title": title,
"content": content,
"embedding": embedding,
"document": pdf_bytes, # No S3 needed
"image": thumbnail, # No S3 needed
"metadata": {"source": "docs", "version": 2}
})
# One query returns everything - vectors + metadata + raw content
results = ds.query("""
SELECT title, content, image, metadata
FROM knowledge_base
WHERE metadata->>'source' = 'docs'
ORDER BY cosine_similarity(embedding, :q)
LIMIT 5
""")What You Gain
- One system: No sync bugs, no orphaned files, no ID mismatches
- Faster retrieval: No S3 round-trip - data is co-located with vectors
- GPU-accelerated search: Orders of magnitude faster than pgvector at scale
- Serverless: Scale to zero, ~200ms provisioning, pay only for usage
- Postgres-compatible: Your SQL knowledge transfers directly
- Branch-per-agent: Multi-agent concurrency without collisions