Storage architecture for physical AI and robotics training data at scale.

TLDR: Physical AI programs (robotics, autonomy, embodied agents, sim-to-real) cross petabyte scale within quarters, across multi-camera video, LiDAR, IMU, joint telemetry, commands, and sim episodes. Traditional lakehouses stall on small-file streaming and can't version or vector-search across modalities.

Use Deeplake as the single tensor-native dataset layer. Backed by S3 / GCS / Azure, chunked for GPU streaming, versioned like Git, and queryable with hybrid vector + scalar, so training, sim, curation, and safety review all read the same bytes at petabyte scale.

What "at scale" means for physical AI data

Physical AI dataset: Millions of multi-minute episodes across modalities, RGB, depth, LiDAR, radar, IMU, joint state, actions, rewards, labels, task metadata, collected from real robots and simulators. Commonly 1–100+ PB per program, growing daily from fleet telemetry.

At this scale, the storage layer is the bottleneck. Every access pattern, training, labeling, sim replay, safety review, regression analysis, stresses a different dimension, and any layer that can't stream, version, or query across modalities slows the whole program.

What a petabyte-scale physical AI dataset needs

Five properties, all at once, at all scales:

• Tensor-native chunking: All modalities stored as typed tensor columns with chunk-level parallel read, not rows of blob URIs.
• Streaming to distributed training: Batches stream to 8–1024 GPUs across regions, with deterministic shuffle and sharding; no disk staging.
• Git-like dataset versioning: Branches, commits, and diffs so label revisions, sim variants, and release candidates are reproducible.
• Hybrid search across modalities: "Find episodes like this failure, in rainy conditions, with task=pick-place" in one query.

Options at petabyte scale

Cost, throughput, ops burden, honestly.

Reference architecture

Fleet + sim feed one versioned dataset. Every downstream consumer reads from it.

Real fleet ──► edge upload ─┐
                             ├─► Deeplake (tensor-native, versioned, S3/GCS/Azure)
Simulation cluster ──────────┘        │
                                      │
                   ┌──────────────────┼──────────────────┐
               Training         Curation             Safety / ops
          (distributed GPU)    (label branches)    (scalar + vector)

The dataset layer is the only source of truth. Training reads from it, sim writes to it, curation branches it, safety queries it. No ETL between systems means no drift between systems.

Stand up a petabyte-capable layer

Three commands get you running on your existing bucket.

1. Install

pip install deeplake

2. Initialize dataset on your bucket

ds = deeplake.create('s3://my-physical-ai/main', schema=ROBOT_SCHEMA)

3. Stream shuffled batches to training

loader = ds.pytorch(batch_size=256, num_workers=16, shuffle=True, distributed=True)

Where teams lose months at scale

• Local-disk cache per node: Works at 10 TB, dies at 1 PB. Caches are never warm after a schema or label change.
• A vector store bolted on later: Two indexes to keep consistent across fleet uploads. They won't be.
• Versioning by folder naming: v3_final_fixed_fixed2. Eventually no one knows which version trained the released policy.
• Sim and real in separate stacks: Different formats means two loaders, two curation pipelines, and two safety reviews. Collapse to one.

FAQ

Does this scale to 100+ PB?

Yes. Deeplake is backed by object storage (S3 / GCS / Azure), which is effectively unbounded. Deeplake's chunk layout is designed for parallel read at high fan-out.

Can we keep our existing S3 buckets?

Yes. Deeplake materializes datasets as objects in your bucket. You keep ownership, IAM, and replication.

What about PII and data governance?

Row- and column-level access control is supported. Deeplake can mask or drop specific columns (e.g. raw video) for analysts who shouldn't see them.

Does it handle 100 Hz sensor data?

Yes. High-frequency signals are stored as tensor time series and sliced at query time by timestamp window.

Can we query across sim and real episodes?

Yes. They share a schema, so queries run across both unless you filter on a source flag.

Is there a managed service?

Yes. Activeloop runs a managed Deeplake that handles chunk lifecycle, replication, and query, letting your team focus on the robots, not the data plane.

Citations

• Activeloop. Deeplake on GitHub.
• NVIDIA. Physical AI reference architectures.
• MCAP, ROS 2 serialization format.

One dataset layer for every robot your fleet produces

Deeplake is tensor-native, versioned, and built for the petabyte scale physical AI reaches fast.

Try Deeplake