Sections in this lesson
Design a Document Ingestion Platform
Build an asynchronous document-to-index pipeline with versioned jobs, isolated parsing and reliable deletion.
Reviewed: · Content owner: Interview Preparator
What you will learn
- Define trust boundaries and component responsibilities.
- Estimate capacity and cost from explicit assumptions.
- Explain retry, failure and recovery behavior.
Before you start
The flow
- 1.Authorize upload
- 2.Persist document job
- 3.Extract in sandbox
- 4.Chunk and embed
- 5.Validate version
- 6.Publish or quarantine
Architecture and boundaries
Private upload store
Tenant-scoped raw objects, trusted ACLs and document revisions.
Stage queues and workers
Isolated parsers, bounded OCR, embedding batches and idempotent stage results.
Version manifest and reconciler
Atomic publication, tombstones and deletion across indexes and caches.
Design assumptions
These values are example sizing inputs, not measurements or capacity guarantees.
| Variable | Assumption |
|---|---|
| Documents per day | 10,000 |
| Chunks per document | 20 |
| Dimensions / bytes per value | 768 / 4 |
Accept uploads and establish ownership
Authorize tenant and destination before issuing a short-lived upload grant. Keep raw documents private and store checksum, media type, document version and trusted ACL metadata separately from document text. Quarantine unsupported or suspicious files.
Use a durable job record and outbox or reconciliation process to close the object-created/job-created gap. Object notifications can be duplicated or reordered. Deduplicate by stable document version and pipeline stage rather than assuming one event means one execution.
Bound extraction and embedding stages
Run parsers without broad credentials or unrestricted network access. Bound decompressed bytes, pages, CPU time and output size; a small archive can expand into a large workload. Preserve source offsets and parser version for traceable chunks.
Separate extraction and embedding queues so slow OCR cannot monopolize embedding capacity. Retry transient failures with a budget; quarantine deterministic failures and expose a reason. Namespace indexes by embedding model/version and dimensions, and apply tenant/document authorization filters at retrieval.
Publish versions and reconcile deletion
Assume 10,000 documents/day, 20 chunks/document and 768 float32 dimensions. This produces 200,000 vectors/day and about 0.6144 GB/day of raw vector values. Text, metadata, index overhead, replicas and old versions are additional; peak demand and OCR-heavy documents determine queue capacity.
Write new-version chunks into a staging namespace, then atomically promote a manifest after validation. Keep a monotonically increasing document revision and deletion tombstone: late workers must not revive old content. Reconcile removal from raw storage, indexes and caches; report deletion progress instead of claiming immediate global erasure.
Try it
{ "jobKey": [ "tenantId", "documentId", "version", "stage" ], "stages": [ "extract", "chunk", "embed", "index", "publish" ], "maxFileBytes": 25000000, "maxPages": 500, "parserNetworkAccess": false, "publishRequiresCurrentRevision": true, "rejectTombstonedDocuments": true}Illustrative design contract, not a deployable service configuration. Validate limits with load and failure tests.
Interview questions
Why not deduplicate only by filename?
Names can repeat or point to new content. Include trusted tenant, stable document ID, version and stage.
How can a deleted document return?
A delayed worker can publish stale chunks unless revision checks and tombstones reject its write.
Are raw vector bytes the total storage cost?
No. Add text, metadata, index overhead, replication and retained versions.
Practice
Trace a dependency outage and a duplicate request; identify admission, durable state and recovery steps.
Check your solution
- Limits and capacity assumptions are explicit.
- Retries do not duplicate effects or results.
- Observability and recovery policy are defined.
Official references
Continue learning
Category sequence
System Design
Lesson 21 / 21
Was this lesson useful?
Change notes: Added AI platform design interview case.