Upload, fast indexing and batch indexing

An uploaded file is not yet searchable evidence. Follow processing to its final state and distinguish delays from failures.

These steps refer to features identified in the application code. This material has not been validated on a live instance; check the documentation and the options in your version.

How it works, step by step

Good documents come before good answers

Prepare readable text, clear headings, tables with headers and sources without unnecessary duplicates. Chunking divides a document into separately retrievable passages; overlap preserves continuity between neighbors. A short chunk can lose conditions, while a large one introduces noise. Metadata preserves source, position and version, plus access scope where enforced by the implementation. Upload stores the file; indexing makes it searchable. When a source changes, check that old chunks disappear and new ones are retrieved before declaring the knowledge base current.

Errors need explicit paths

A timeout differs from a permission error. Retry only transient errors, with bounded attempts and increasing delays. An operation with side effects may execute before its response is lost; retrying without an idempotency key can duplicate the effect. Streaming shows progress but does not guarantee completion. Checkpoints may allow failed stages to resume within runtime limits. Store the run identifier, stage, duration and error while removing secrets from logs. Measure slow paths as well as averages.

Cost belongs to the whole path

Count model calls, input and output tokens, embeddings, reranking, vision and tools. Reasoning can consume billable tokens without an equally long visible response. Indexing costs and conversation costs differ. Use current prices rather than a permanent number from a course. A simple estimate is tokens/1,000,000 × price plus additional operations; application credits may have their own conversion. Compare cost per successful task, not just per call. Parallel execution can reduce latency while increasing total consumption. A budget needs a verified stopping condition.

The visual map

Follow the solid arrows for the main flow. Dashed blue arrows supply data or context; dashed pink arrows show feedback or returning results. Colors and shapes distinguish models, stores, decisions and outputs. On smaller screens, scroll horizontally to follow the entire diagram.

A complete example

Upload the policy in RAG. Fast starts ordinary processing; batch may defer provider operations. Check notifications and document status before testing a question.

Try it yourself

1. Upload a synthetic document and follow upload, processing and query testing.
2. Record the input, source and expected outcome before running the experiment. Use only the fictional data in the example.
3. Follow the diagram stages. At every step record what information is received and produced; do not confuse intermediate output with the final outcome.
4. Repeat after removing necessary information or making the input ambiguous. Check whether the system clarifies, stops or invents an answer.
5. Compare with the explained solution. Keep the configuration, date, result and an explanation for differences. Change one thing and retest.

An explained solution

The document is searchable only after successful processing; retain error stage and message for diagnosis. A successful exercise lets you show the connection between input, stages and outcome. When information is missing, a cautious answer is more useful than invented details. Compare more than style: check conditions, sources and operations too.

When it helps and what can go wrong

Check format and size limits in the current version. Choose this approach when it improves a measured need. Keep a simple baseline and compare outcomes using identical inputs. One successful example does not establish reliability in every situation.

Check your understanding

Does completed upload mean a document is ready for RAG?

No. Extraction, chunking and indexing must also complete successfully.

Does a timeout prove the operation did not execute?

No. Check state before retrying an action with side effects.

What outcome should this exercise produce?

The document is searchable only after successful processing; retain error stage and message for diagnosis.

Words to remember

• Chunk: A document passage that can be retrieved separately.
• Idempotency: Controlled repetition without duplicating an effect.
• Latență / Latency: Time until a useful result.

Sources and your next step

• Lewis et al. — Retrieval-Augmented Generation
• LangGraph — Workflows and agents
• OpenRouter — Reasoning tokens

To prepare: D01 — Preparing documents for AI