Skip to content

Indexing pipeline

This guide describes what happens when you call CodebaseIndexer.index().

Overview

The indexer performs a full scan of a repository commit and writes files, chunks, and relations into PostgreSQL. It does not generate embeddings; embeddings are handled by the separate embed() step.

Steps

  1. Repository registration

  2. PostgresGraphStorage.ensure_repository() creates or updates the repository row.

  3. Git sync and snapshot creation

  4. GitVolumeManager updates a bare mirror of the repository.
  5. create_snapshot() inserts a snapshot with status indexing.

  6. If the same commit is already indexed, the existing snapshot is reused.

  7. File scan

  8. The worktree is scanned with os.walk.
  9. Ignored directories: .git, node_modules, __pycache__, .venv, dist, build, target, vendor.

  10. Extensions indexed:

    • .py, .js, .jsx, .ts, .tsx, .java, .go, .rs, .c, .cpp, .php, .html, .css

  11. Parallel parsing

  12. A ProcessPoolExecutor processes file chunks (50 files per task, 5 workers).

  13. Each worker uses TreeSitterRepoParser to:

    • Skip large files (>1 MB), binaries, and minified/generated content.
    • Emit FileRecord, ChunkNode, ChunkContent, and child_of relations.
    • Build FTS documents from chunk metadata and content.

  14. Snapshot activation

  15. Indexing stats and a file manifest are generated.
  16. The snapshot is marked completed and becomes the active snapshot.

Notes

  • If a repository is already being indexed, index() returns "queued".
  • Semantic tags from Tree-sitter queries are currently available for Python, JavaScript, and TypeScript.
  • Parsing and indexing always run on the full file set for the target commit.