# Obuild Design Documentation

This document explains the architecture and design decisions of obuild, a parallel, incremental, and declarative build system for OCaml.

## Table of Contents

1. [Genesis and Philosophy](#genesis-and-philosophy)
2. [Architectural Layers](#architectural-layers)
3. [Core Data Flow](#core-data-flow)
4. [The Two-DAG Architecture](#the-two-dag-architecture)
5. [Dependency Resolution System](#dependency-resolution-system)
6. [Build Execution Pipeline](#build-execution-pipeline)
7. [Type System Design](#type-system-design)
8. [Ctypes.cstubs Integration](#ctypescstubs-integration)
9. [Custom Code Generators](#custom-code-generators)
10. [PPX and Preprocessor Resolution](#ppx-and-preprocessor-resolution)
11. [Configuration Files](#configuration-files)
12. [CLI Reference](#cli-reference)
13. [Design Decisions and Trade-offs](#design-decisions-and-trade-offs)

---

## Genesis and Philosophy

Obuild started on a bank holiday after xmas, as an experiment to make the
simplest OCaml build system. The main goals are to:

* provide a good user experience.
* provide a building black box, no mocking around generic rules.
* provide features in the highest level possible.
* the cleanest build possible, with by-products hidden from the user.
* provide good defaults, and standardize names as much as possible.
* expose everything that the user of the build system needs in one place.
* be simple to build to prevent any bootstrapping problem.

One of the main influences was Haskell Cabal, which provides to all Haskellers
a simple way to provide a build system to a project with a single file. This
applies well for the myriad of OCaml options too.

### Simple to Build

Obuild is buildable with just the compiler and the compiler standard library.
This make bootstrapping very easy: all you need is the OCaml compiler installed.

This creates some pain for developers of obuild, as lots of basic functions
available in others libraries need to written again as part of obuild. As the
initial development was done really quickly, some functions are not as
performant (CPU or memory-wise) as they could be. This can be fixed as problem
becomes apparent in scaling.

### Simple to Use

Each project is described really simply in a one place, in a user friendly format.
A central .obuild file is used, and provides high level description of your project.
Along with some meta data (name, authors, description, etc), it defines the library,
and/or executable that the project wants to have, from which inputs (source
files, modules).

All dependencies are fully autogenerated internally and used to recompile only
the necessary bits.

Along with library and executable, test and benchmark can be defined, so as to
provide an easy way to test or bench some part of your project. It also provides
a standard on how to build and execute tests and benchmarks.

### Design Principles

```
┌─────────────────────────────────────────────────────────────────┐
│                     OBUILD DESIGN GOALS                         │
├─────────────────────────────────────────────────────────────────┤
│  Declarative    │ Users describe targets, not build steps      │
│  Parallel       │ Independent tasks run concurrently           │
│  Incremental    │ Only rebuild what changed                    │
│  Self-contained │ No external build tool dependencies          │
│  Discoverable   │ Auto-detect modules, dependencies            │
└─────────────────────────────────────────────────────────────────┘
```

---

## Architectural Layers

Obuild is organized in four distinct layers, each with clear responsibilities:

```
┌─────────────────────────────────────────────────────────────────┐
│                        LAYER 4: CLI                             │
│                       src/main.ml                               │
│              Command dispatcher, user interface                 │
├─────────────────────────────────────────────────────────────────┤
│                    LAYER 3: COMMANDS                            │
│            lib/help.ml, init.ml, install.ml, etc.              │
│           High-level operations (init, install, doc)            │
├─────────────────────────────────────────────────────────────────┤
│                      LAYER 2: CORE                              │
│                     lib/core/*.ml                               │
│    Parsing, analysis, dependency resolution, build execution    │
├─────────────────────────────────────────────────────────────────┤
│                      LAYER 1: BASE                              │
│                     lib/base/*.ml                               │
│         Filepath, Filesystem, Fugue (utilities), CLI            │
└─────────────────────────────────────────────────────────────────┘
```

### Layer 1: Base Utilities (`lib/base/`)

The foundation layer has **no dependencies** on OCaml build tools. This is critical for bootstrapping - obuild must compile itself before it can use itself.

| Module | Purpose |
|--------|---------|
| `compat.ml` | OCaml version compatibility shims |
| `filepath.ml` | Type-safe path abstractions |
| `filesystem.ml` | File I/O operations |
| `fugue.ml` | Functional utilities (string, list, option) |
| `cli.ml` | Command-line parsing framework |

### Layer 2: Core Build System (`lib/core/`)

This layer contains ~60 modules implementing the build system logic:

```
lib/core/
├── Parsing ──────────── obuild_lexer.ml, obuild_parser.ml,
│                        obuild_ast.ml, obuild_validate.ml,
│                        project.ml, project_read.ml, meta.ml
│
├── Types ────────────── types.ml, target.ml, libname.ml,
│                        modname.ml, hier.ml, filetype.ml
│
├── Analysis ─────────── analyze.ml, prepare.ml, prepare_types.ml,
│                        dependencies.ml, metacache.ml
│
├── Execution ────────── build.ml, scheduler.ml, taskdep.ml,
│                        process.ml, prog.ml, buildprogs.ml
│
├── Configuration ────── configure.ml, gconf.ml, findlibConf.ml
│
└── Utilities ────────── dag.ml, dagutils.ml, utils.ml, helper.ml
```

### Layer 3: Command Modules (`lib/`)

High-level operations that users invoke:

| Module | Command | Purpose |
|--------|---------|---------|
| `help.ml` | `obuild help` | Documentation system |
| `init.ml` | `obuild init` | Create new projects |
| `install.ml` | `obuild install` | Install artifacts |
| `doc.ml` | `obuild doc` | Generate documentation |
| `sdist.ml` | `obuild sdist` | Create source distributions |

### Layer 4: Main Entry Point (`src/main.ml`)

The CLI dispatcher that:
- Parses command-line arguments
- Dispatches to appropriate command handler
- Manages global configuration
- Handles top-level error reporting

---

## Core Data Flow

The build process follows a clear pipeline from configuration to compiled artifacts:

```
┌──────────────────────────────────────────────────────────────────────────┐
│                           BUILD PIPELINE                                  │
└──────────────────────────────────────────────────────────────────────────┘

  .obuild file                    META files              Source files
       │                              │                        │
       ▼                              ▼                        ▼
┌─────────────┐              ┌─────────────────┐      ┌───────────────┐
│   PARSING   │              │   DEPENDENCY    │      │    MODULE     │
│             │              │   RESOLUTION    │      │   ANALYSIS    │
│ Lexer       │              │                 │      │               │
│ Parser      │              │ FindlibConf     │      │ ocamldep      │
│ Validator   │              │ Meta parser     │      │ Hier mapping  │
│             │              │ Metacache       │      │               │
└──────┬──────┘              └────────┬────────┘      └───────┬───────┘
       │                              │                        │
       ▼                              ▼                        ▼
┌──────────────────────────────────────────────────────────────────────────┐
│                              ANALYSIS                                     │
│                           (analyze.ml)                                    │
│                                                                          │
│   Combines all inputs into project_config with resolved dependencies      │
└──────────────────────────────────────────────────────────────────────────┘
                                    │
                                    ▼
┌──────────────────────────────────────────────────────────────────────────┐
│                             PREPARATION                                   │
│                            (prepare.ml)                                   │
│                                                                          │
│   For each target: scan modules, build compilation DAG, compute paths     │
└──────────────────────────────────────────────────────────────────────────┘
                                    │
                                    ▼
┌──────────────────────────────────────────────────────────────────────────┐
│                              EXECUTION                                    │
│                             (build.ml)                                    │
│                                                                          │
│   Scheduler runs compile_steps in parallel, respecting dependencies       │
└──────────────────────────────────────────────────────────────────────────┘
                                    │
                                    ▼
                          Compiled Artifacts
                     (dist/build/<target>/...)
```

### Sequence Diagram: Full Build

```
User          main.ml       Project_read    Analyze      Prepare       Build       Scheduler
  │               │              │             │            │            │             │
  │──build───────>│              │             │            │            │             │
  │               │──read()─────>│             │            │            │             │
  │               │              │──parse──────│            │            │             │
  │               │              │──validate───│            │            │             │
  │               │<──Project.t──│             │            │            │             │
  │               │              │             │            │            │             │
  │               │──prepare()───────────────>│            │            │             │
  │               │              │             │──resolve───│            │             │
  │               │              │             │  META      │            │             │
  │               │              │             │  files     │            │             │
  │               │<──project_config───────────│            │            │             │
  │               │              │             │            │            │             │
  │               │──build_dag()────────────────────────────────────────>│             │
  │               │              │             │            │            │             │
  │               │              │             │──prepare_target()──────>│             │
  │               │              │             │<──compilation_state─────│             │
  │               │              │             │            │            │             │
  │               │              │             │            │──schedule()────────────>│
  │               │              │             │            │            │    ┌────────┤
  │               │              │             │            │            │    │parallel│
  │               │              │             │            │            │    │compile │
  │               │              │             │            │            │    │steps   │
  │               │              │             │            │            │    └────────┤
  │               │              │             │            │<───────────────done──────│
  │<──success─────│              │             │            │            │             │
```

---

## The Two-DAG Architecture

A distinctive feature of obuild is its use of **two separate DAGs** (Directed Acyclic Graphs) for dependency tracking. Understanding why requires understanding two different questions:

### Question 1: "What changed?"

The **Files DAG** answers this question. It tracks file-level dependencies:

```
Files DAG Example (for module Bar that uses Foo):

                    bar.cmo
                   /   |   \
                  /    |    \
             bar.ml  bar.cmi  foo.cmi
                       |
                    bar.mli
```

- **Nodes**: Individual files (.ml, .mli, .cmi, .cmo, .cmx, .c, .h, .o)
- **Edges**: "file A depends on file B" (based on content)
- **Purpose**: Check modification times to determine what needs rebuilding

When `foo.ml` changes but `foo.mli` doesn't:
- `bar.cmo` doesn't need rebuilding (it only depends on `foo.cmi`)
- Only `foo.cmo` needs recompilation

### Question 2: "What order?"

The **Steps DAG** answers this question. It tracks task execution order:

```
Steps DAG Example:

              LinkTarget(mylib)
                /    |    \
               /     |     \
    CompileModule  CompileModule  CompileC
       (Bar)          (Foo)       (stubs.c)
         |              |
    CompileInterface  CompileInterface
       (Bar)            (Foo)
```

- **Nodes**: Compilation tasks (CompileModule, CompileInterface, CompileC, LinkTarget)
- **Edges**: "task A must complete before task B"
- **Purpose**: Enable parallel scheduling while respecting dependencies

### Why Two DAGs?

| Aspect | Files DAG | Steps DAG |
|--------|-----------|-----------|
| Purpose | Incremental builds | Parallel execution |
| Answers | "What changed?" | "What order?" |
| Granularity | Individual files | Compilation tasks |
| Usage | mtime comparison | Topological sort |

Separating these concerns allows:
1. **Fine-grained incrementality**: Only recompile files whose dependencies changed
2. **Maximum parallelism**: Independent tasks run concurrently
3. **Correct ordering**: Dependent tasks wait for prerequisites

---

## Dependency Resolution System

Obuild resolves dependencies through OCamlfind's META file system. This is one of the more complex subsystems.

### Internal META Parsing

ocamlfind is the current de-facto standard for installed package querying.
ocamlfind is usually injected on the command line to ocamlopt, ocamldep, ocamlc
with special flags (-syntax, -package), that ocamlfind will re-write to call
the program with something that the program can understand. All the information
for this transformation is stored in META files.

Unfortunately this design prevents META caching, and each time ocamlc/ocamlopt
is used it will reparse the META files. This also causes problems if ocamlfind
does not exist when used as a program, or if the library is not installed when
used as a library.

Because of those 2 reasons, obuild has its own implementation of META parsing
with caching support.

### META File Discovery

```
Search Order for library "base":

1. Check each path in OCAMLPATH:
   ├── /home/user/.opam/default/lib/base/META     ← Found!
   ├── /usr/lib/ocaml/base/META
   └── ...

2. Alternative: META.<name> format:
   └── /usr/lib/ocaml/META.base
```

### META Parsing Flow

```
                     META file (text)
                           │
                           ▼
                    ┌─────────────┐
                    │   LEXER     │
                    │ (meta.ml)   │
                    └──────┬──────┘
                           │ tokens
                           ▼
                    ┌─────────────┐
                    │   PARSER    │
                    │ (meta.ml)   │
                    └──────┬──────┘
                           │
                           ▼
                    ┌─────────────┐
                    │   Pkg.t     │
                    │  (cached)   │
                    └─────────────┘
```

### Metacache Architecture

A critical design decision is how META files are cached:

```
┌─────────────────────────────────────────────────────────────────┐
│                        METACACHE                                 │
│                                                                 │
│   Hashtbl: main_name → (filepath, root_pkg)                     │
│                                                                 │
│   "base" → ("/path/to/base/META", <Pkg.t for base>)            │
│   "str"  → ("/path/to/str/META", <Pkg.t for str>)              │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘
                              │
                              │ get_from_cache("base.shadow_stdlib")
                              ▼
              Returns: (filepath, ROOT package for "base")
                              │
                              │ Caller must resolve subpackages:
                              │ Meta.Pkg.find(["shadow_stdlib"], root)
                              ▼
                    Resolved subpackage Pkg.t
```

**Key design**: Metacache always returns the **root package**. Callers are responsible for navigating to subpackages. This prevents:
- Double resolution (cache pre-resolving, then caller resolving again)
- Incorrect path generation for subpackages

### Include Path Resolution

META files specify directories using special syntax:

```
Directory Field      Resolves To
─────────────────────────────────────────────────
""  or "."           basePath (where META lives)
"^"                  parent directory
"^subdir"            parent/subdir
"+stdlib"            $OCAMLLIB/stdlib
"/absolute/path"     /absolute/path
"relative/path"      basePath/relative/path
```

### Dependency Graph Construction

```
Analyze.prepare() builds multiple graphs:

┌─────────────────────────────────────────────────────────────────┐
│                    project_pkgdeps_dag                          │
│              (Complete dependency picture)                       │
│                                                                 │
│    ┌─────────┐         ┌─────────┐         ┌─────────┐         │
│    │Exe:main │────────>│Lib:mylib│────────>│Dep:base │         │
│    └─────────┘         └────┬────┘         └────┬────┘         │
│                             │                    │               │
│                             ▼                    ▼               │
│                        ┌─────────┐         ┌─────────┐         │
│                        │Dep:unix │         │Dep:sexp │         │
│                        └─────────┘         └─────────┘         │
└─────────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────────┐
│                    project_targets_dag                          │
│              (Internal targets only)                            │
│                                                                 │
│         ┌─────────┐                   ┌─────────┐              │
│         │Exe:main │──────────────────>│Lib:mylib│              │
│         └─────────┘                   └─────────┘              │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────────┐
│                     project_dep_data                            │
│              (Dependency classification)                        │
│                                                                 │
│         mylib  →  Internal   (defined in project)              │
│         base   →  System     (from OCamlfind)                  │
│         unix   →  System     (from OCamlfind)                  │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘
```

---

## Build Execution Pipeline

### Scheduler Design

The scheduler manages parallel job execution with dependency tracking:

```
┌─────────────────────────────────────────────────────────────────┐
│                         SCHEDULER                                │
│                                                                 │
│   ┌─────────────┐    ┌─────────────┐    ┌─────────────┐        │
│   │   Ready     │    │   Running   │    │  Completed  │        │
│   │   Queue     │───>│   Jobs      │───>│   Tasks     │        │
│   └─────────────┘    └─────────────┘    └─────────────┘        │
│         ▲                                      │                │
│         │                                      │                │
│         └──────────────────────────────────────┘                │
│              When dependencies are satisfied                    │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘
```

### Compile Steps

Each step represents an atomic compilation task:

```ocaml
type compile_step =
  | CompileModule    of Hier.t    (* .ml → .cmo/.cmx *)
  | CompileInterface of Hier.t    (* .mli → .cmi *)
  | CompileDirectory of Hier.t    (* Pack directory modules *)
  | CompileC         of filename  (* .c → .o *)
  | GenerateCstubsTypes     of Libname.t  (* ctypes type discovery *)
  | GenerateCstubsFunctions of Libname.t  (* ctypes stub generation *)
  | CompileCstubsC          of Libname.t  (* Compile generated C *)
  | LinkTarget       of target    (* Link library/executable *)
  | CheckTarget      of target    (* Verify outputs exist *)
```

### Build Execution Sequence

```
For library "mylib" with modules Foo, Bar (Bar depends on Foo):

Time ──────────────────────────────────────────────────────────>

Thread 1:  ┌──────────────────┐
           │CompileInterface  │
           │     (Foo)        │
           └────────┬─────────┘
                    │
           ┌────────▼─────────┐
           │  CompileModule   │
           │     (Foo)        │
           └────────┬─────────┘
                    │
                    ▼ (Bar can now start)
           ┌──────────────────┐
           │  CompileModule   │
           │     (Bar)        │
           └────────┬─────────┘
                    │
                    ▼
           ┌──────────────────┐
           │   LinkTarget     │
           │    (mylib)       │
           └──────────────────┘

Thread 2:  ┌──────────────────┐
           │CompileInterface  │  (parallel with Foo interface)
           │     (Bar)        │
           └──────────────────┘
                    │
                    │ (waits for CompileModule(Foo))
                    ▼
           ┌──────────────────┐
           │    CompileC      │  (parallel with Bar module)
           │   (stubs.c)      │
           └──────────────────┘
```

---

## Type System Design

Obuild uses domain-specific types to prevent common errors:

### Path Types

```ocaml
(* lib/base/filepath.ml *)

type filepath   (* Directory or file path *)
type filename   (* Just a filename, no directory *)

(* These are abstract types - you cannot mix them up *)
val (</>) : filepath -> filename -> filepath  (* Combine path + name *)
val (<//>) : filepath -> filepath -> filepath (* Combine paths *)
```

### Name Types

```
┌─────────────────────────────────────────────────────────────────┐
│                        NAME TYPES                                │
│                                                                 │
│  Libname.t                                                      │
│  ├── main_name: "base"                                         │
│  └── subnames: ["shadow_stdlib"]                               │
│      → "base.shadow_stdlib"                                    │
│                                                                 │
│  Modname.t                                                      │
│  └── Validated OCaml module name                               │
│      → "Base" (capitalized, valid chars)                       │
│                                                                 │
│  Hier.t                                                         │
│  └── Module hierarchy: [Base; Utils; String_extra]             │
│      → "Base.Utils.String_extra" (module path)                 │
│      → "base/utils/string_extra" (file path)                   │
│                                                                 │
│  Target.Name.t                                                  │
│  ├── Lib of Libname.t                                          │
│  ├── Exe of string                                             │
│  ├── Test of string                                            │
│  ├── Bench of string                                           │
│  └── Example of string                                         │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘
```

### Why This Matters

```ocaml
(* WRONG: Easy to confuse in stringly-typed code *)
let path = dir ^ "/" ^ name ^ ".ml"  (* What if name has a slash? *)

(* RIGHT: Types prevent mistakes *)
let path = dir </> (fn (name ^ ".ml"))  (* Type-checked! *)
```

---

## Ctypes.cstubs Integration

Obuild supports ctypes.cstubs for generating C bindings declaratively.

### Configuration Syntax

```
library mylib
  modules: Bindings, C, Types_generated
  build-deps: ctypes, ctypes.stubs

  cstubs
    external-library-name: mylib_stubs
    type-description: Bindings.Types -> Types_gen
    function-description: Bindings.Functions -> Funcs_gen
    generated-types: Types_generated
    generated-entry-point: C
    headers: string.h
```

### Cstubs Build Flow

```
                         .obuild configuration
                                 │
                                 │ cstubs block parsed
                                 ▼
┌─────────────────────────────────────────────────────────────────┐
│                    PHASE 1: Type Discovery                       │
│                                                                 │
│   Generate discover.ml ───> Compile ───> Run ───> discover.c   │
│                                                    │            │
│   Compile discover.c ───> Run ───> types_generated.ml          │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘
                                 │
                                 ▼
┌─────────────────────────────────────────────────────────────────┐
│                  PHASE 2: Compile Bindings                       │
│                                                                 │
│   bindings.ml (user's functor definitions)                      │
│        │                                                        │
│        ▼                                                        │
│   Compile bindings.cmo                                          │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘
                                 │
                                 ▼
┌─────────────────────────────────────────────────────────────────┐
│                  PHASE 3: Stub Generation                        │
│                                                                 │
│   Generate stubgen.ml ───> Compile with bindings.cmo ───> Run  │
│        │                                                        │
│        ├───> mylib_stubs_generated.ml (FOREIGN implementation) │
│        ├───> c.ml (entry point)                                │
│        └───> mylib_stubs.c (C stubs)                           │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘
                                 │
                                 ▼
┌─────────────────────────────────────────────────────────────────┐
│                  PHASE 4: Compile Everything                     │
│                                                                 │
│   Compile generated .ml files                                   │
│   Compile mylib_stubs.c ───> mylib_stubs.o                     │
│   Archive ───> libmylib_stubs.a                                │
│   Link everything into final library                            │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘
```

### Cstubs DAG Integration

```
                    LinkTarget(mylib)
                   /      |       \
                  /       |        \
    CompileModule    CompileCstubsC    CompileModule
    (C - entry)      (stubs.c)      (Mylib_stubs_generated)
         |               |                   |
         └───────────────┼───────────────────┘
                         │
              GenerateCstubsFunctions
                         │
              ┌──────────┴──────────┐
              │                     │
    CompileModule(Bindings)   GenerateCstubsTypes
                                    │
                         CompileModule(Types_generated)
```

---

## Custom Code Generators

Obuild supports custom code generators defined in the `.obuild` file. This allows any code generation tool (menhir, ocamllex, atdgen, protobuf, etc.) to be integrated without hardcoding support in obuild itself.

### Generator Architecture

Generators are registered during project parsing and invoked during module resolution. When obuild looks for a module and cannot find a `.ml` file, it checks for source files matching registered generator suffixes.

```
Module resolution for "Parser":

  1. Look for parser.ml in src-dir       → Not found
  2. Check registered generators:
     ├── suffix "mly" → look for parser.mly  → Found!
     └── suffix "mll" → look for parser.mll  → Not found
  3. Run menhir generator on parser.mly
  4. Produces parser.ml (and parser.mli)
  5. Continue compilation with generated files
```

### Generator Types

Generators are defined in `lib/core/generators.ml` with two key types:

```ocaml
(* Built-in generator *)
type t = {
  suffix : string;                    (* File extension to match *)
  modname : Modname.t -> Modname.t;   (* Module name transformation *)
  commands : filepath -> filepath -> string -> string list list;
  generated_files : filename -> string -> filename;
}

(* Custom generator from .obuild file *)
type custom = {
  custom_name : string;               (* Unique identifier *)
  custom_suffix : string option;      (* Auto-detection suffix *)
  custom_command : string;            (* Command template *)
  custom_outputs : string list;       (* Output file patterns *)
  custom_module_name : string option; (* Optional module name pattern *)
}
```

### Variable Substitution

Generator commands support template variables:

| Variable | Description |
|----------|-------------|
| `${src}` | Full path to source file |
| `${dest}` | Destination path without extension |
| `${base}` | Base filename without extension |
| `${srcdir}` | Source directory |
| `${destdir}` | Destination directory |
| `${sources}` | Space-separated list of all inputs (multi-input only) |

### Generator Execution Flow

```
.obuild file                  Module resolution           Build execution
     │                              │                          │
     │  generator menhir            │  Parser not found        │
     │    suffix: mly               │  parser.mly exists       │
     │    command: menhir ...       │                          │
     │    outputs: ${base}.ml       │                          │
     │                              │                          │
     ▼                              ▼                          ▼
register_custom()           get_generators("mly")      run(dest, src, modName)
     │                              │                          │
     │  Stored in global            │  Returns matching        │  substitute_variables()
     │  generator registry          │  generators              │  Execute via sh -c
     │                              │                          │  Produce .ml/.mli
```

Custom generators defined in `.obuild` take precedence over built-in ones, allowing users to override default behavior (e.g., using menhir instead of ocamlyacc for `.mly` files).

---

## PPX and Preprocessor Resolution

The `ppx_resolver.ml` module handles resolution of PPX preprocessors and legacy camlp4 syntax extensions.

### Resolution Strategy

PPX and syntax packages are resolved through the META file system using special predicates:

```
For a target with syntax dependencies:

  1. Collect syntax deps from project DAG
  2. For each syntax dep:
     ├── Internal (project library)?
     │     └── Use compiled bytecode archive path
     └── External (findlib package)?
           └── Query META with predicates:
               [Syntax, Preprocessor, Camlp4o/Camlp4r]
  3. Generate -I include paths
  4. Generate archive file arguments
  5. Return complete preprocessor command
```

### Key Functions

- `get_syntax_pp`: Generates preprocessor flags (`-I` paths and archive files) for syntax packages
- `get_target_pp`: Resolves all syntax/preprocessor dependencies for a target and returns a complete preprocessor configuration

The resolver distinguishes between internal syntax packages (compiled as part of the project) and external ones (resolved via findlib META files).

---

## Configuration Files

Obuild supports configuration files (`.obuildrc`) for setting default values that apply before command-line argument parsing.

### File Locations

Configuration is loaded from two locations, in order:

1. `~/.obuildrc` — User-level defaults
2. `./.obuildrc` — Project-level defaults

Project-level values override user-level values. Command-line arguments override both.

### File Format

```
# Comments start with #
verbose = true
jobs = 4
ocamlopt = /usr/local/bin/ocamlopt
```

Each line is a `key = value` pair. Keys correspond to CLI option names (without the `--` prefix).

### Integration

The CLI framework (`lib/base/cli.ml`) loads configuration files via `load_config()` and applies them as argument defaults through `run_with_config`. This happens before command-line parsing, so CLI flags always take precedence.

```
Priority (highest to lowest):

  1. Command-line arguments     --jobs 8
  2. Project .obuildrc          jobs = 4
  3. User ~/.obuildrc           jobs = 2
  4. Built-in defaults          jobs = 2
```

---

## CLI Reference

### Commands

| Command | Description |
|---------|-------------|
| `configure` | Detect dependencies and prepare build configuration |
| `build` | Compile targets (all or specified) |
| `clean` | Remove build artifacts |
| `install` | Install compiled artifacts |
| `test` | Build and run test targets |
| `init` | Create a new project skeleton |
| `sdist` | Create a source distribution tarball |
| `get` | Query configuration values |
| `generate` | Generate helper files (see subcommands below) |
| `doc` | Generate documentation (not yet implemented) |
| `infer` | Infer project structure (not yet implemented) |

### Generate Subcommands

| Subcommand | Description |
|------------|-------------|
| `generate merlin` | Create `.merlin` file for IDE support (source dirs, build dirs, packages) |
| `generate opam` | Create `.opam` file (OPAM 2.0 format) from project metadata |
| `generate completion` | Generate shell completion scripts (bash, zsh, fish) |

### Global Options

These options apply to all commands:

| Option | Short | Description |
|--------|-------|-------------|
| `--help` | `-h` | Show help |
| `--version` | `-V` | Show version |
| `--verbose` | `-v` | Enable verbose output |
| `--quiet` | `-q` | Suppress output |
| `--debug` | `-d` | Enable debug output |
| `--debug+` | | Enable debug output with commands |
| `--color` | | Enable colored output |
| `--findlib-conf` | | Path to findlib configuration |
| `--ocamlopt` | | Path to ocamlopt compiler |
| `--ocamldep` | | Path to ocamldep tool |
| `--ocamlc` | | Path to ocamlc compiler |
| `--cc` | | Path to C compiler |
| `--ar` | | Path to ar archiver |
| `--pkg-config` | | Path to pkg-config tool |
| `--ranlib` | | Path to ranlib tool |
| `--ocamldoc` | | Path to ocamldoc tool |
| `--ld` | | Path to linker |

### Configure Options

| Option | Default | Description |
|--------|---------|-------------|
| `--executable-native` | true | Build native executables |
| `--executable-bytecode` | false | Build bytecode executables |
| `--executable-profiling` | false | Build profiling executables |
| `--executable-debugging` | false | Build debugging executables |
| `--executable-as-obj` | false | Build executables as objects |
| `--library-native` | true | Build native libraries |
| `--library-bytecode` | true | Build bytecode libraries |
| `--library-profiling` | false | Build profiling libraries |
| `--library-debugging` | false | Build debugging libraries |
| `--library-plugin` | true (Unix) | Build library plugins |
| `--build-examples` | false | Build example targets |
| `--build-benchs` | false | Build benchmark targets |
| `--build-tests` | false | Build test targets |
| `--annot` | false | Generate annotation files |

### Build Options

| Option | Short | Description |
|--------|-------|-------------|
| `--jobs` | `-j` | Number of parallel jobs (default: 2) |
| `--dot` | | Dump dependency graph in DOT format |
| `--noocamlmklib` | | Disable ocamlmklib usage |
| `-g` | | Shorthand for `--library-debugging --executable-debugging` |

---

## Design Decisions and Trade-offs

### Decision 1: Two Separate DAGs

**Trade-off**: More complex implementation vs. better incrementality and parallelism.

**Rationale**: A single DAG cannot efficiently answer both "what changed?" (file-level) and "what order?" (task-level). Separating them allows:
- File DAG: Fine-grained mtime checks
- Steps DAG: Coarse-grained parallel scheduling

### Decision 2: Metacache Returns Root Packages

**Trade-off**: Callers must resolve subpackages vs. no double-resolution bugs.

**Rationale**: When the cache pre-resolved subpackages, callers that also resolved (common pattern) caused `SubpackageNotFound` errors. Returning root packages consistently prevents this class of bugs.

### Decision 3: Type-Safe Path Abstractions

**Trade-off**: More verbose code vs. compile-time path safety.

**Rationale**: Path manipulation bugs are common and hard to debug. Type-safe `filepath`/`filename` types catch errors at compile time rather than runtime.

### Decision 4: Exception-Based Error Handling

**Trade-off**: Less explicit control flow vs. simpler code structure.

**Rationale**: OCaml's exception model allows errors to propagate naturally without Result types threading through every function. The main entry point catches and formats all exceptions.

### Decision 5: Lazy META Parsing with Caching

**Trade-off**: First build is slower vs. subsequent builds are faster.

**Rationale**: Parsing META files is I/O bound. Caching parsed results in memory dramatically speeds up dependency resolution for large projects with many dependencies.

### Decision 6: New Parser Architecture

**Trade-off**: More code, separate modules vs. cleaner separation of concerns.

**Rationale**: The new parser architecture (Lexer → Parser → AST → Validator) separates:
- Tokenization (obuild_lexer.ml)
- Syntax analysis (obuild_parser.ml)
- Data representation (obuild_ast.ml)
- Semantic validation (obuild_validate.ml)

This makes each component easier to test, modify, and understand.

---

## Summary

Obuild's architecture reflects its goals of being declarative, parallel, and incremental:

```
┌─────────────────────────────────────────────────────────────────┐
│                     ARCHITECTURE SUMMARY                         │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│   Declarative Input        Parallel Execution                   │
│        │                         │                              │
│   .obuild file             Scheduler + DAGs                     │
│        │                         │                              │
│        ▼                         ▼                              │
│   ┌─────────┐              ┌─────────┐                         │
│   │ Parsing │──────────────│ Build   │                         │
│   │ Layer   │   Analysis   │ Layer   │                         │
│   └─────────┘              └─────────┘                         │
│        │                         │                              │
│   Project.t               Compiled Artifacts                    │
│        │                         │                              │
│   Type-Safe               Incremental Rebuilds                  │
│   Representation                                                │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘
```

The key insight is that build systems solve two distinct problems:
1. **What to build**: Declarative configuration, dependency resolution
2. **How to build**: Scheduling, parallelism, incrementality

Obuild addresses both through its layered architecture and dual-DAG design.

---

## Librification

Obuild has been designed to be used as a library eventually.
The code is shifting towards using pure structures and functions, so
that things can be reused. There is some global state that will be
eventually reduced to provide better control of each part.

One possible development would be to provide an optional daemon
that monitors file changes and automatically rebuilds on demand without having
to re-analyze the whole project.

Some other possible scenarios include having other programs use the project file
format, either to provide tools to write them or tools that read them.