# FlexLock Usage Guide

A practical, end-to-end tour of the library. Each section answers one
question: *what does this give me, and how do I get it?*

## Table of Contents

1. [Overview](#1-overview)
2. [Installation & first run](#2-installation--first-run)
3. [Configuration model](#3-configuration-model)
4. [Submitting work — three flavors](#4-submitting-work--three-flavors)
5. [`py2cfg` — build configs from Python](#5-py2cfg--build-configs-from-python)
6. [Sweeps](#6-sweeps)
7. [Snapshots & reproducibility](#7-snapshots--reproducibility)
8. [Smart run / caching](#8-smart-run--caching)
9. [HPC backends](#9-hpc-backends)
10. [MLflow integration](#10-mlflow-integration)
11. [Debugging](#11-debugging)
12. [Exceptions](#12-exceptions)
13. [Environment variables](#13-environment-variables)
14. [Common recipes](#14-common-recipes)
15. [FAQ / gotchas](#15-faq--gotchas)

---

## 1. Overview

FlexLock is a thin orchestration layer around OmegaConf and Python's
`multiprocessing` / Slurm / PBS. It runs the same function you'd write
anyway, and gives you three things in return:

1. **A reproducible `save_dir`.** Every submission resolves `save_dir`
   eagerly (so resolvers like `${vinc:}` advance exactly once) and writes
   `run.lock` before the function runs.
2. **A snapshot of the environment.** Git tree hash for tracked repos,
   xxh64 of tracked data files, the resolved config — all in `run.lock`.
3. **A cache.** When you submit the same fingerprint again, FlexLock
   returns the cached result instead of re-executing.

It sits between Hydra (config composition) and a job scheduler (Slurm,
PBS, local multiprocessing): you write the function, FlexLock handles
config merging, save-dir lifecycle, snapshotting, parallel dispatch, and
cache lookup.

---

## 2. Installation & first run

```bash
pip install flexlock
# or
conda install -c quentinf00 flexlock
```

Minimal "hello world":

```python
# train.py
from flexlock import flexcli

@flexcli
def train(lr: float = 0.01, epochs: int = 10, save_dir: str = 'outputs/train'):
    """Train a model."""
    print(f"lr={lr}, epochs={epochs}, save_dir={save_dir}")
    return {"accuracy": 0.95}

if __name__ == "__main__":
    train()
```

```bash
python train.py -o lr=0.1
```

After the run, `outputs/train/` contains:

```
outputs/train/
├── run.lock         # config + git tree hash + data hashes, written before exec
├── run.complete     # written after the function returns; cache-hit marker
├── results.json     # the function's return value (if dict-shaped)
└── snapshot/        # filtered git tree if you snapshot repos / data
```

`run.lock` is the canonical record. `run.complete` is what
`smart_run` looks for to decide a run is a real cache hit (a run with
only `run.lock` was interrupted).

---

## 3. Configuration model

FlexLock configs are OmegaConf `DictConfig`s. A config that should *run*
something carries a `_target_` key whose value is a dotted import path:

```yaml
# config.yaml
_target_: myproject.train.train
lr: 0.01
epochs: 100
save_dir: outputs/train
```

At submit time, FlexLock recursively instantiates any nested dict that
has `_target_`, then calls the top-level target with the remaining keys
as keyword arguments.

### Defaults files

A "defaults" file exports a dict (or `DictConfig`) under a named
variable. Three forms are accepted by `-d` and `Project(defaults=...)`:

```bash
# Dotted module path — variable is the last segment
flexlock-run -d myproject.config.defaults

# File path with explicit variable
flexlock-run -d configs/defaults.py:defaults

# Bare file path — defaults to the `defaults` variable
flexlock-run -d configs/defaults.py
```

The file-based forms register the loaded file in `sys.modules` under
its file stem, so any `_target_` strings captured by `py2cfg` inside the
file remain importable when workers re-import them later. `flexlock-run`
also inserts the current working directory into `sys.path`, so dotted
imports of project-root packages work without `PYTHONPATH=.`.

### Overrides and selection

The merge order for `flexlock-run` is:

```
defaults  +  -c yaml  +  -m merge_file  +  -o root.overrides
           → -s select_node
           +  -M post-select merge  +  -O post-select overrides
           → execute / sweep
```

`-O` and `-M` apply *after* selection, so you can override the selected
node without disturbing the root.

### Swapping a subtree: `key=@name`

A value starting with `@` names a config instead of giving one, and
**replaces** the node (the way Hydra config groups do), rather than merging
into it:

```bash
flexlock-run -d my.xps.train_x -s main -o main.lit_module.model=@big_model
flexlock-run -d my.xps.train_x -s main -O lit_module.model=@big_model   # after select
flexlock-run -d my.xps.train_x -s main -o main.model=@other.module.huge_model
flexlock-run -d my.xps.train_x -s main --sweep "@small_model,@big_model" \
    --sweep-target main.lit_module.model
```

- `@name` is an attribute of the `-d` module (`my.xps` above), so building
  blocks defined next to your presets (`small_model = py2cfg(...)`) can be
  swapped by name. `@pkg.module.attr` reaches a config defined elsewhere.
- Replacement matters: merging `big_model` into `small_model` would keep the
  keys only `small_model` has. With `@`, the node is exactly `big_model`.
- Overrides apply in order, so `-o main.model=@big_model main.model.width=16`
  swaps, then tweaks the new model.
- `@@x` is the literal string `@x`. An unknown name is an error listing the
  module's config attributes.
- The override is recorded as typed in `_preset_`; the expanded subtree is in
  `config`, so provenance is complete. The same works from Python:
  `Project("my.xps.train_x").submit("main", overrides={"model": "@big_model"})`
  and in sweep items (`{model: "@big_model"}` in a sweep file).

This replaces registry resolvers such as `${sel_model:small}` over a
`_MODELS` dict.

### Relative interpolations

Within a node, prefer relative refs over root-scope refs. They're the
only kind of cross-key reference that survives `proj.get()` *and*
follows per-item sweep overrides:

```python
# myproject/configs.py
from flexlock import py2cfg
from myproject.train import train

defaults = dict(
    train=py2cfg(
        train,
        save_dir='outputs/train',
        log_dir='${.save_dir}/logs',         # sibling key — relative
        ckpt_dir='${.save_dir}/checkpoints', # sibling key — relative
    ),
)
```

`${.x}` resolves against the current node, `${..x}` against its parent,
`${...x}` against its grandparent. These are preserved by `proj.get()`
and by sweep merging, so per-item `save_dir` overrides correctly
propagate into `log_dir` and `ckpt_dir`. Root-scope refs (`${root_anchor}`,
`${main.save_dir}`) are *frozen to their concrete value* at `proj.get()`
time, which keeps the returned sub-tree pickleable for HPC dispatch.

### `save_dir_policy` — collision guard (replaces `${vinc:}` / `${now:}`)

Keep `save_dir` a **stable** path in configs; `save_dir_policy` decides what
happens when it's already occupied (default `"raise"`; `"increment"` versions
it the way `${vinc:}` used to, `"timestamp"` replaces `${now:}`). See
[`resolvers.md`](resolvers.md#save_dir_policy-replaces-vinc--now) for the full
policy table and migration guide. `${vinc:}`/`${now:}` still work for one
deprecation cycle but emit a `DeprecationWarning`.

```python
save_dir='outputs/train/run'
proj.submit(cfg, save_dir_policy="increment")
# → outputs/train/run_0000, run_0001, ...
```

Other resolvers: `${latest:glob_pattern}` (newest match),
`${run_lock:<run_dir>,<dotted.key>}` (read a value from an upstream
`run.lock`).

---

## 4. Submitting work — three flavors

### 4a. CLI via `@flexcli` + `flexlock-run`

For scripts you'll mostly run from the shell:

```python
# train.py
from flexlock import flexcli

@flexcli
def train(lr: float = 0.01, save_dir: str = 'outputs/train'):
    """Train a model."""
    return {"accuracy": 0.95}

if __name__ == "__main__":
    train()
```

```bash
python train.py -o lr=0.1
flexlock-run -d train.train -O lr=0.1     # equivalent
flexlock-run -d train.train --note "baseline before lr sweep"   # record intent
flexlock-run --help                        # prints argparse help + compiled config
```

You get an `ExecutionResult.result` back (the dict), `run.lock`,
`run.complete`, and `results.json` on disk.

Pass `--note` (CLI) or `note="…"` (`proj.submit(cfg, note=…)`) to record a
free-text intent in `run.lock`. It never affects caching and shows up in
`flexlock show`/`graph`/`report`.

### 4b. Python `Project().submit(cfg)`

For multi-stage scripts and notebooks where you want pipeline plumbing
(`proj.get`, `proj.defaults`, `proj.run_stage`, ...):

```python
from flexlock import Project

proj = Project(defaults='myproject.configs.defaults')

cfg = proj.get('train')
cfg.lr = 0.1
result = proj.submit(cfg)

print(result.save_dir)         # 'outputs/train'
print(result.status)           # 'SUCCESS' or 'CACHED'
print(result['accuracy'])      # dict-like access to result.result
```

`proj.get(key)` returns a self-contained sub-tree: root-scope refs are
frozen, resolver calls and intra-node refs are preserved.

### 4c. `from flexlock import submit` — one-shot

For when you already have a `DictConfig` and don't need a Project:

```python
from flexlock import submit, py2cfg
from myproject.train import train

cfg = py2cfg(train, lr=0.01, save_dir='outputs/train')
result = submit(cfg, slurm_config='configs/slurm_gpu.yaml')
```

`submit` is `Project().submit` with all the same kwargs (`sweep`,
`smart_run`, `overrides`, `force`, `isolated`, ...). Use it for ad-hoc
one-offs and library code that takes a config and runs it.

---

## 5. `py2cfg` — build configs from Python

`py2cfg` reads a function/class signature and produces a config dict
keyed by `_target_`:

```python
from flexlock import py2cfg
from myproject.train import train
from myproject.models import Transformer

cfg = py2cfg(
    train,
    lr=0.01,
    model=py2cfg(Transformer, layers=12, heads=16),
    save_dir='outputs/train',
)
# {
#   '_target_': 'myproject.train.train',
#   'lr': 0.01,
#   'epochs': 10,                       # captured from signature default
#   'model': {'_target_': 'myproject.models.Transformer', 'layers': 12, 'heads': 16},
#   'save_dir': 'outputs/train',
# }
```

Use it when:
- you want to refactor a function and have its config follow (signature
  changes propagate automatically),
- you want nested instantiation (`model=py2cfg(...)`),
- you want IDE autocomplete and type-checking on the keyword names.

Prefer YAML when configs are written by non-Python users, or when you
want a declarative record that lives outside the codebase.

**Gotchas:**
- The target must be re-importable on workers. `py2cfg(train)` inside a
  notebook produces `_target_: __main__.train`, which won't import in a
  spawned subprocess. Move definitions to a module.
- Every default in the signature ends up in the config, including
  `None` defaults. Drop them via overrides if you don't want them in
  `run.lock`.
- Decorated functions are unwrapped automatically (`py2cfg` looks at
  `_original_fn`).

---

## 6. Sweeps

A sweep is a list of override dicts merged into the base config one at
a time. Sources:

```python
proj.submit(cfg, sweep=[{'lr': 0.001}, {'lr': 0.01}])              # inline
proj.submit(cfg, sweep_file='configs/grid.yaml')                    # not directly — see below
flexlock-run -d ... --sweep-file configs/grid.yaml
flexlock-run -d ... --sweep "0.001,0.01,0.1" --sweep-target lr
flexlock-run -d ... --sweep-key param_grid                          # list lives in defaults
```

From Python, use `load_sweep` to read a file:

```python
from flexlock import load_sweep, Project

proj = Project(defaults='myproject.configs.defaults')
sweep = load_sweep(sweep_file='configs/grid.yaml')

if __name__ == "__main__":
    results = proj.submit('train', sweep=sweep, n_jobs=4)
    for r in results:
        print(r.save_dir, r['accuracy'])
```

### `sweep_target`

Without `sweep_target`, each item is merged at the root of the base
config. With `sweep_target='optimizer.lr'`, scalar items are placed at
that dotted path.

### `sweep_dir_suffix`

When `True`, each item's `save_dir` becomes
`<base.save_dir>/sweep_{i:04d}/` — items are **nested under** the
sweep root, not siblings to it. This keeps the tasks DB and lineage
markers inside the same tree:

```python
proj.submit(cfg, sweep=sweep, sweep_dir_suffix=True)
# outputs/train/sweep_0000/
# outputs/train/sweep_0001/
# outputs/train/run.lock.tasks.db
```

Containment is validated up-front: if a sweep item's `save_dir` falls
outside the sweep root, you get a `FlexLockValidationError` listing the
offenders before any work is queued.

### Enqueue a sweep for later (`--sweep` + `--enqueue`)

`--enqueue queue.yaml` normally appends the single compiled config to a queue
file. Combined with a sweep source it appends **one entry per item**, each
already merged at `--sweep-target` and frozen (deferred resolvers preserved):

```bash
flexlock-run -d ... --sweep "0.001,0.01,0.1" --sweep-target lr --enqueue q.yaml
# q.yaml now has 3 entries; run them later (locally or on HPC) with:
flexlock-run -d ... --sweep-file q.yaml --n_jobs 3
```

This is the single-stage counterpart of the multi-stage `--enqueue` batching in
§14g — there each item is a composite `{_stages_: [...]}` pipeline task instead
of one plain config.

### Per-item interpolations

Because relative refs (`${.x}`, `${..x}`) survive node selection and
sweep merging, a sweep that overrides `save_dir` automatically updates
sibling keys that point to it:

```python
from flexlock import py2cfg, Project
from myproject.train import train

cfg = py2cfg(
    train,
    save_dir='outputs/train',
    log_dir='${.save_dir}/logs',          # follows per-item save_dir
    ckpt_dir='${.save_dir}/ckpts',
)

if __name__ == "__main__":
    proj = Project()
    proj.submit(cfg, sweep=[
        {'save_dir': 'outputs/sweep/a'},  # log_dir → outputs/sweep/a/logs
        {'save_dir': 'outputs/sweep/b'},  # log_dir → outputs/sweep/b/logs
    ], n_jobs=2)
```

### `n_jobs`, `isolated`, and the `__main__` guard

- `n_jobs=1` (default, no `isolated`): runs in-process. No guard needed.
- `n_jobs > 1` or `isolated=True`: uses `multiprocessing` with the
  `spawn` start method (avoids CUDA fork hazards). Spawn re-imports the
  launching script in each child, so **any module-scope
  `proj.submit(...)` call must sit under `if __name__ == "__main__":`**
  or the children will recursively re-launch.
- Notebooks and REPL contexts are fine.

After a parallel or `isolated=True` run, workers write
`results.json` into each per-task `save_dir`. The driver loads them
back and populates `ExecutionResult.result`, so the return value is
available downstream just like in serial execution.

---

## 7. Snapshots & reproducibility

Every submit with a `save_dir` writes `run.lock` with three sections:

```yaml
# outputs/train/run.lock
timestamp: "2026-06-01T14:30:00"
repos:
  myproject:
    commit: "abc123…"
    tree: "def456…"               # this is what smart_run compares
    is_dirty: false
    path: "/abs/path/to/repo"
data:
  train_data: "xxh64_789xyz…"
config:
  _target_: myproject.train.train
  lr: 0.01
  save_dir: outputs/train
```

### Declaring what to track

```python
from flexlock import py2cfg
from myproject.train import train

cfg = py2cfg(
    train,
    input_path='data/train.csv',
    save_dir='outputs/train',
    _snapshot_=dict(
        repos={
            'myproject': '.',                       # string shorthand → {'path': '.'}
            'mylib': {'path': 'libs/mylib'},        # explicit
            'pkg':   {'module': 'pkg'},             # resolved via importlib
            'filtered': {                           # filtered comparison
                'path': '.',
                'include': ['src/model/**'],
                'exclude': ['tests/**'],
            },
        },
        data={'train': '${...input_path}'},
        prevs=['${...upstream_run_dir}'],           # link to upstream runs
    ),
)
```

**Relative refs inside `_snapshot_` need three dots.** They resolve from
their own position in the tree, and `data` values and `prevs` items both
sit two levels below the stage node (`stage._snapshot_.data.train`), so
climbing back to a sibling of `_snapshot_` takes `${...key}`:
`data.train` → `data` → `_snapshot_` → stage. A single-dot `${.input_path}`
would look for `input_path` *inside* the `data` mapping and raise
`InterpolationKeyError`.

When a config has `_target_`, FlexLock auto-tracks the git repo of the
target's source file — you don't need to spell `repos` for the main
codebase. Add explicit entries for sibling libraries, vendored
dependencies, or filtered comparisons.

### `snapshot()` directly

You can also write a snapshot outside the `submit` flow:

```python
from flexlock import snapshot

snapshot(cfg, repos={'main': '.'}, data={'train': 'data/train.csv'})
```

### Re-running from a snapshot

`run.lock` is YAML — pull the `config` block out and resubmit it:

```python
from omegaconf import OmegaConf
from flexlock import submit

cfg = OmegaConf.load('outputs/train/run.lock').config
submit(cfg, force=True)   # re-execute, overwriting in place
```

`force=True` invalidates `run.complete` (so `smart_run` doesn't short-
circuit), preserves outputs and `run.lock`, and re-runs the function.

### Presets: which config produced a run

A **preset** is what you pass on the command line: the `-d` target (a Python
attribute holding a config) plus the `-s` key. You don't register anything:
every run made through `flexlock-run -d/-s`, or through
`Project("pkg.module.attr").get(key)`, records its preset in the config as a
reserved `_preset_` key, along with the overrides you typed:

```yaml
# run.lock
config:
  _target_: sst_ml_mapping.starter.train.train
  _preset_:
    defaults: sst_ml_mapping.starter.xps_glob:train_small_cloud_gap_compact
    select: main
    overrides: ["main.lit_module.lr=1e-4"]   # -o, as typed
    overrides_after_select: []               # -O
    merges: []                               # -m / -M
  ...
```

`_preset_` is provenance only: it never enters the fingerprint (so caching is
unaffected), `RunDiff` ignores it, and `instantiate` strips it before calling
your function. Only runnable nodes (`_target_` + `save_dir`) get one.

When a run completes, FlexLock adds a symlink under the project's
`.flexlock/presets/<defaults>/<select>/`, which makes the runs of a preset
cheap to list:

```bash
flexlock runs xps_glob.train_small_cloud_gap_compact       # newest first
flexlock runs train_small_cloud_gap_compact -s main --strict  # no overrides
flexlock presets sst_ml_mapping.starter.xps_glob             # the catalogue
flexlock presets sst_ml_mapping.starter.xps_glob --format md > docs/presets.md
```

`flexlock presets` lists every attribute of a config module that holds a
runnable stage (at its top level or one level down; deeper nodes such as
models or callbacks are building blocks), with the comment block above its
definition and its number of runs. It can replace a hand-maintained index of
commands. Addresses match by suffix: `xps_glob.train_x`, `xps_glob:train_x`
and `train_x` all find `pkg.starter.xps_glob:train_x`.

Put a `.flexlock/` directory at the project root (the project template does)
so one set of links serves every results directory; without it the links live
next to each results directory's index, which `flexlock runs` also finds from
the project root. `flexlock reindex <results_root>` rebuilds the links. Runs
made before presets were recorded have no `_preset_` and don't appear.

### Referencing another preset's runs: `${run:}`

To use the output of another experiment, reference its preset instead of
copying a versioned path:

```python
infer_cloud_gap = dict(
    train_dir="${run:xps_glob.train_small_cloud_gap,main}",   # newest complete run
    main=py2cfg(infer_fn, xp_path="${train_dir}", ...),
)
```

Add a pin (`${run:xps_glob.train_x,main,0005}`) to use a specific version, or
`strict` (`${run:xps_glob.train_x,main,strict}`) to ignore runs made with
overrides. The reference is resolved when you submit: the chosen path is
logged, frozen into the config (and into `--dump`/`--enqueue` output), and
added to the run's lineage. Keep quick tests from being picked up by making
them their own preset (`train_x_debug = ...`). Full reference:
[resolvers](resolvers.md).

---

## 8. Smart run / caching

```python
proj.submit(cfg, smart_run=True, search_dirs=['outputs/train/'])
```

Mechanism:

1. Compute a stable **fingerprint** from `cfg` (per-repo git *tree* hashes
   restricted to include/exclude, data hashes, and the resolved config with
   `save_dir` prefix-normalized so output location doesn't affect it). This
   uses `git write-tree` only — no commits or refs are created.
2. Look the fingerprint up in the project-wide index (one indexed `SELECT`).
   The index maps a fingerprint to where a completed run lives — a `run.lock`
   directory **or** a sweep task `(task_db, task_id)`, so sweep items are
   first-class cache entries.
3. On a hit, verify the pointed-to run still exists and is complete
   (`run.complete`, or the task's `results.json`). A stale pointer self-prunes
   and is treated as a miss. Only `status='done'` runs are ever served.
4. On an index miss, fall back to the legacy `**/run.lock` glob scan +
   `RunDiff` (controlled by `FLEXLOCK_INDEX_FALLBACK`, default on) and backfill
   the index on a hit, so the slow path self-eliminates.

Rebuild the index at any time with `flexlock reindex <results_root>`; deleting
it is always safe.

The index lives in the nearest `.flexlock/` directory above the run (walking up
from its parent), or `<parent>/.flexlock/` if there is none; `$FLEXLOCK_INDEX`
overrides both. A `.flexlock/` at the project root therefore collects every
run in the project, which is what `flexlock runs` and cache lookups from the
project root expect. The same directory holds the preset links (§7).

### When caches miss

- The git tree hash changed (any tracked file modified, including
  untracked-but-not-ignored files).
- A tracked data path's xxh64 changed.
- The config changed (after resolution, ignoring `_snapshot_`).
- The candidate run never wrote `run.complete`.

To debug a miss:

```bash
flexlock-diff dirs outputs/train/run_0001 outputs/train/run_0002
# exit code: 0 = match, 1 = differ, 2 = error; add --format json for scripting
flexlock why outputs/train/run_0001 outputs/train/run_0002   # + the commits between them
```

### Failure records (`run.error`)

When a run's user function raises, FlexLock writes a `run.error` JSON sidecar next
to `run.lock` (`exc_type`, `exc_message`, full `traceback`, timestamp; plus
`task_id`/`node` for sweep tasks). The original exception still propagates
unwrapped — capture never masks it. Triage without opening sqlite:

```bash
flexlock show outputs/train/run_0001 --format json    # .status == "failed", .error.*
```

A successful re-run (`force=True` / `--force`) clears the stale `run.error`. See
[Agentic Workflows](agentic_workflows.md) for the full schema and triage guidance.

### Knobs

| Knob                                | Effect                                     |
|-------------------------------------|--------------------------------------------|
| `smart_run=False`                   | Always execute                             |
| `force=True`                        | Invalidate `run.complete` (and, for sweeps, per-item markers + the task DB), then execute |
| `FLEXLOCK_NO_CACHE=1`               | Disable the on-disk data-hash cache        |
| `FLEXLOCK_INDEX=/path/index.db`     | Explicit fingerprint-index location        |
| `FLEXLOCK_INDEX_FALLBACK=0`         | Skip the legacy glob scan on an index miss |
| `match_include` / `match_exclude`   | Override git path filters at compare time  |
| `search_dirs=[...]`                 | Where to look for cached runs              |

Combine with sweeps: re-running a 32-job sweep with `smart_run=True`
re-executes only the ones whose fingerprints don't match an existing
completed run.

---

## 9. HPC backends

Slurm and PBS are supported via YAML config files. The same path works
for `submit` (single job) and `submit(..., sweep=..., n_jobs=N)` (array
job).

```yaml
# configs/slurm_gpu.yaml
startup_lines:
  - "#SBATCH --job-name=flexlock"
  - "#SBATCH --gres=gpu:1"
  - "#SBATCH --cpus-per-task=8"
  - "#SBATCH --mem=32G"
  - "#SBATCH --time=04:00:00"
  - "#SBATCH --array=0-31"
  - "module load cuda/11.8"
  - "source activate myenv"
python_exe: "python"
```

```python
proj.submit(cfg, sweep=sweep, slurm_config='configs/slurm_gpu.yaml')
```

```yaml
# configs/pbs_gpu.yaml — container example
startup_lines:
  - "#PBS -l select=1:ncpus=8:mem=32gb"
  - "#PBS -l walltime=04:00:00"
  - "#PBS -J 0-31"
  - "cd $PBS_O_WORKDIR"
python_exe: |
  singularity run --nv
  --bind $(pwd):/workspace --pwd /workspace
  env.sif python
```

`wait=False` submits and returns immediately. `wait=True` blocks until
the array finishes; the default timeout comes from
`FLEXLOCK_DEFAULT_TIMEOUT` (seconds).

Preview the rendered submission script without submitting:

```python
proj.submit(cfg, slurm_config='configs/slurm_gpu.yaml', dry_run=True)
# Prints the generated Slurm script plus any validation warnings.
```

### Multi-stage pipelines on HPC

A multi-stage selection runs on HPC too: array workers fan out **across sweep
items**, and each item's stages run in order on the worker that claims it.

```bash
flexlock-run -d project.pipeline.pipeline_cfg -s train linear_probe \
    --sweep-target pipeline_dir --sweep results/xp1,results/xp2 \
    --slurm-config configs/slurm_gpu.yaml
```

From Python this is `Project.submit_pipeline(items, slurm_config=...)`, where
`items[i]` is the list of frozen stage configs for item `i`. Because all stages
of an item share one scheduler job, size the job for the heaviest stage (see the
one-job-per-item limitation in §14g). `dry_run=True` renders the script without
queueing anything.

Anything that goes wrong on the backend side — bad YAML, queue
rejection, task-DB lock contention — is raised as
`FlexLockBackendError`.

---

## 10. MLflow integration

`mlflow_context` wraps an MLflow run that's bound to a physical
`save_dir`. It implements an "always new + deprecate old" strategy so
re-runs into the same directory don't pollute the previous MLflow run:

```python
from flexlock import flexcli, mlflow_context
import mlflow

@flexcli
def train(lr: float = 0.01, save_dir: str = 'outputs/train'):
    with mlflow_context(save_dir, experiment_name='my-exp') as run:
        mlflow.log_metric('train_loss', 0.32)
        mlflow.log_metric('val_accuracy', 0.95)
    return {"accuracy": 0.95}

if __name__ == "__main__":
    train()
```

The context inherits user tags from the previously active run for the
same directory, so pipeline tags like `pipeline_run=run_001` persist
across stages without manual plumbing. MLflow not installed? The
context is a no-op and yields `None`.

---

## 11. Debugging

### `@debug_on_fail`

```python
from flexlock import debug_on_fail

@debug_on_fail
def train(...):
    ...
```

On exception, the decorator either drops you into PDB (script context)
or injects the function's locals into the caller's scope (notebook
context). Strategy is chosen by `FLEXLOCK_DEBUG_STRATEGY`:

| Value        | Behavior                                           |
|--------------|----------------------------------------------------|
| `auto` (default) | PDB for scripts, locals injection for notebooks |
| `pdb`        | Always PDB post-mortem                             |
| `inject`     | Always inject locals                               |

### Activation

Two control surfaces, two semantics:

- **`@flexcli` / `flexlock-run --debug`:** opt *in* to wrapping the user
  function with `debug_on_fail`. `FLEXLOCK_DEBUG=1` (or `--debug`)
  enables it.
- **Manually applied `@debug_on_fail`:** active by default. To turn it
  off (e.g. in production), set `FLEXLOCK_NODEBUG=1` — the decorator
  returns the original function unchanged. Note: `FLEXLOCK_DEBUG=0` does
  *not* disable a manually applied decorator.

The decorator accepts (and warns about) legacy kwargs like
`stack_depth=`, so old snippets won't blow up — they're just ignored.

### `print_config`

```bash
flexlock-run -d configs/defaults.py -s train -O lr=0.1 --print-config
```

Prints the fully-merged config plus the target function's docstring,
then exits. From Python:

```python
proj.submit('train', overrides={'lr': 0.1}, print_config=True)
# When sweep is also passed, prints one config per sweep item.
```

`flexlock-run --help` prints argparse help followed by the same
compiled config view.

### Logging to file

```python
from pathlib import Path
from flexlock import flexcli, log_to_file
from loguru import logger

@flexcli
def train(lr: float = 0.01, save_dir: str = 'outputs/train'):
    with log_to_file(Path(save_dir) / 'debug.log'):
        logger.debug(f"starting with lr={lr}")
        ...
```

`log_to_file` is a context manager that adds a Loguru sink for its
lifetime and removes it on exit. Importable from the package root.

---

## 12. Exceptions

All FlexLock exceptions inherit from `FlexLockError`.

| Exception                  | Wraps                                                                 |
|----------------------------|-----------------------------------------------------------------------|
| `FlexLockConfigError`      | Invalid config structure, bad `_target_`, malformed `_snapshot_`     |
| `FlexLockValidationError`  | Sweep `save_dir` containment, ambiguous sweep source, missing select |
| `FlexLockSnapshotError`    | Can't write `run.lock`, git/data hash failures                       |
| `FlexLockCacheError`       | Cache read / fingerprint comparison failures                          |
| `FlexLockExecutionError`   | FlexLock-side dispatch failures (NOT user exceptions — see below)    |
| `FlexLockBackendError`     | Slurm/PBS submission failures, queue errors, task DB lock issues     |

**Important:** exceptions raised *inside the user's target function*
are **not** wrapped. They propagate as the original type
(`ValueError`, `RuntimeError`, ...). Catch them directly:

```python
from flexlock import FlexLockBackendError

try:
    result = proj.submit(cfg, slurm_config='configs/slurm_gpu.yaml')
except FlexLockBackendError as e:
    print(f"HPC failed: {e}")
except ValueError as e:
    print(f"User function raised: {e}")
```

---

## 13. Environment variables

| Variable                       | Default                  | Effect                                                  |
|--------------------------------|--------------------------|---------------------------------------------------------|
| `FLEXLOCK_DEBUG`               | `false`                  | Opt into debug wrapping via `@flexcli` / `flexlock-run` |
| `FLEXLOCK_NODEBUG`             | `false`                  | Disable a manually applied `@debug_on_fail`             |
| `FLEXLOCK_DEBUG_STRATEGY`      | `auto`                   | `auto` / `pdb` / `inject`                               |
| `FLEXLOCK_NO_CACHE`            | `false`                  | Disable on-disk data-hash cache                         |
| `FLEXLOCK_CACHE`               | `~/.cache`               | Cache base dir                                          |
| `FLEXLOCK_DEFAULT_N_JOBS`      | `1`                      | Default `--n_jobs` if not given                         |
| `FLEXLOCK_DEFAULT_TIMEOUT`     | `3600`                   | HPC wait timeout (seconds)                              |
| `FLEXLOCK_POLL_INTERVAL`       | `10`                     | HPC poll interval (seconds)                             |
| `FLEXLOCK_WARN_SMART_RUN`      | `true`                   | Warn when `smart_run=True` and `search_dirs=None`       |
| `FLEXLOCK_TIMESTAMP_FORMAT`    | `%Y-%m-%dT%H-%M-%S`      | Directory timestamp format                              |
| `FLEXLOCK_CONFIGURE_LOGGING`   | `true`                   | Configure Loguru on import                              |

Boolean values accept `1`/`true`/`yes`/`on` and `0`/`false`/`no`/`off`.
See [reference.md](./reference.md) for the full list.

---

## 14. Common recipes

### 14a. 16-job hyperparameter sweep on Slurm

```python
# scripts/sweep.py
from flexlock import Project, load_sweep

if __name__ == "__main__":
    proj = Project(defaults='myproject.configs.defaults')
    sweep = [{'lr': lr, 'batch_size': bs}
             for lr in [0.001, 0.01, 0.1, 1.0]
             for bs in [16, 32, 64, 128]]   # 16 items

    results = proj.submit(
        'train',
        sweep=sweep,
        sweep_dir_suffix=True,
        slurm_config='configs/slurm_gpu.yaml',  # set #SBATCH --array=0-15
        smart_run=True,
    )

    best = max(results, key=lambda r: r.get('val_accuracy', 0))
    print(f"best: {best.save_dir} → {best['val_accuracy']}")
```

### 14b. Resume a partially-finished sweep

Re-run the same script. `smart_run=True` skips every completed item
(has both `run.lock` and `run.complete`); only interrupted items
re-execute. No special command needed.

### 14c. Re-run only the runs that failed

```python
from pathlib import Path
from flexlock import Project

proj = Project(defaults='myproject.configs.defaults')

# Items that started but didn't complete
incomplete = [d for d in Path('outputs/sweep').glob('sweep_*/')
              if (d / 'run.lock').exists() and not (d / 'run.complete').exists()]

if __name__ == "__main__":
    for d in incomplete:
        cfg = proj.get('train')
        cfg.save_dir = str(d)
        proj.submit(cfg, force=True)   # re-exec into same dir
```

### 14d. Pull sweep results back as a DataFrame

```python
import pandas as pd

rows = []
for r in results:        # list[ExecutionResult] from proj.submit(..., sweep=...)
    rows.append({
        'save_dir': r.save_dir,
        'lr': r.cfg.lr,
        'batch_size': r.cfg.batch_size,
        'accuracy': r.get('accuracy'),
    })
df = pd.DataFrame(rows)
```

### 14e. Single isolated GPU run

When you want the child's CUDA context to die at exit (so a later
sweep can fork without the parent holding a stale CUDA handle):

```python
from flexlock import submit, py2cfg
from myproject.train import train

cfg = py2cfg(train, lr=0.01, save_dir='outputs/train')

if __name__ == "__main__":
    result = submit(cfg, isolated=True)
    print(result['accuracy'])
```

### 14e-bis. Pick a stage to run with `fzf`

`flexlock stages` lists a defaults tree's runnable stages without resolving the
config (so `${vinc:}` never fires):

```bash
DEF=myproject.pipeline.cfg
flexlock-run -d "$DEF" -s "$(flexlock stages -d "$DEF" --format keys | fzf)"
```

### 14f. Multi-stage pipeline with auto-discovery

```python
from flexlock import Project

proj = Project(defaults='myproject.configs.defaults')

prep_cfg = proj.get('preprocess')
prep_cfg.save_dir = 'outputs/exp_001/preprocess'
proj.run_stage(prep_cfg)

train_cfg = proj.get('train')
train_cfg.data_dir = prep_cfg.save_dir         # propagated by run_stage
train_cfg.save_dir = 'outputs/exp_001/train'
proj.run_stage(train_cfg, isolated=True)

proj.save_snapshot('outputs/exp_001')          # writes pipeline.yaml
```

`run_stage` auto-discovers `search_dirs` by scanning sibling experiment
directories with the same stage name, so re-running with different
hyperparameters cache-hits where it can.

---

### 14g. Experiment-directory pattern (`pipeline_dir` anchor)

A single config declares every stage of a project and nests them all under one
**experiment root** interpolation anchor. Each stage's `save_dir` — and every
cross-stage reference (an upstream checkpoint, a stats file) — is written
relative to that anchor:

```python
# project/pipeline.py
from flexlock import py2cfg

def make_pipeline_cfg():
    return dict(
        pipeline_dir="???",                       # required: the experiment root

        params=dict(lr=1e-4, encoder_size="small", img_size=128),

        train=py2cfg(
            train_stage,
            save_dir="${pipeline_dir}/train",
            lr="${params.lr}",
            encoder=py2cfg(Encoder, size="${params.encoder_size}",
                           img_size="${params.img_size}"),
        ),
        linear_probe=py2cfg(
            probe_stage,
            save_dir="${pipeline_dir}/linear_probe",
            checkpoint="${pipeline_dir}/train/checkpoints/best.ckpt",   # upstream artifact
            encoder=py2cfg(Encoder, size="${params.encoder_size}",
                           img_size="${params.img_size}"),
        ),
    )

pipeline_cfg = make_pipeline_cfg()
```

Run one stage into an experiment; the anchor decides where it lands:

```bash
# train lands in results/myxp/train
flexlock-run -d project.pipeline.pipeline_cfg -s train -o pipeline_dir=results/myxp

# add a stage later against the SAME experiment -> results/myxp/linear_probe,
# sitting next to the training run and reading its checkpoint
flexlock-run -d project.pipeline.pipeline_cfg -s linear_probe -o pipeline_dir=results/myxp

# run one stage across several experiments at once
flexlock-run -d project.pipeline.pipeline_cfg -s linear_probe \
    --sweep pipeline_dir=results/myxp1,results/myxp2

# run several stages of ONE experiment in order (space- or comma-separated)
flexlock-run -d project.pipeline.pipeline_cfg -s train linear_probe \
    -o pipeline_dir=results/myxp

# run several stages across several experiments at once (sweep × stages)
flexlock-run -d project.pipeline.pipeline_cfg -s train linear_probe \
    --sweep-target pipeline_dir --sweep results/myxp1,results/myxp2 --n_jobs 2

# same, on Slurm — array workers fan out across experiments
flexlock-run -d project.pipeline.pipeline_cfg -s train linear_probe \
    --sweep-target pipeline_dir --sweep results/myxp1,results/myxp2 \
    --slurm-config slurm.yaml
```

Passing several stages to `-s` runs them **in order as a pipeline**. Each sweep
item becomes one *composite pipeline task*: its stages run sequentially on a
single worker, so a downstream stage always sees the upstream stage's artifacts
already on disk (that's the whole ordering contract — stages compose through
`${pipeline_dir}/<stage>/...` paths, not in-memory wiring). Parallelism and HPC
array workers fan out **across items**, never within one item. With no sweep,
that's a single composite task running locally in-process.

Each sweep item is merged into the **root** config *before* stage selection
(respecting `--sweep-target` as a root dot-path), then every stage is
re-selected and re-frozen from that merged root — so `-s train linear_probe
--sweep-target pipeline_dir --sweep xp1,xp2` builds 2 items × 2 stages, and each
item's stages land under its own `pipeline_dir`. With `--check-exists`, stages
already completed for a `pipeline_dir` are skipped, so re-running the sequence
resumes where it stopped.

Multi-stage composes with `--sweep*`, `--slurm-config`/`--pbs-config`, and
`--enqueue`. It is only incompatible with `-O`/`-M` and `-e`, which target *the*
selected node (ambiguous across a sequence) — pass shared values as root-level
overrides (`-o pipeline_dir=...`, `-m`, `-c`) instead.

**Batching with `--enqueue`.** `-s train linear_probe --enqueue queue.yaml`
appends one composite `{_stages_: [...]}` entry per sweep item to the queue file
(deferred resolvers like `${run_lock:}`/`${latest:}` are preserved unresolved).
Run the whole queue later — including on HPC — with
`flexlock-run -d ... --sweep-file queue.yaml`; composite entries run as-is (never
re-merged with the base config), and plain entries in the same file are merged
into the compiled node config and run as single-stage pipelines.

**Stage records.** Each executed stage keeps its own config, fingerprint,
lineage, code-drift information, and preset link. By default its directory
contains a full `run.lock`. With `FLEXLOCK_TASK_RECORD=db`, the shared task row
stores snapshots keyed by absolute stage directory; the normal record reader
selects the appropriate stage through its `.flexlock_marker`. Downstream
`${run_lock:...}` references can read an upstream stage before the whole task
finishes. Exporting a composite task includes all stage snapshots with the
master's code and environment information. Compare individual stage
directories with `flexlock-diff`; a composite task ID names multiple records.

**One job per item (accepted limitation).** All stages of an item run inside one
scheduler job / worker slot, so per-stage HPC resource configs (e.g. a CPU
extract stage then a GPU train stage with different `--gres`) are not possible.
Give the whole pipeline the resources its heaviest stage needs, or split the
stages into separate submissions. Scheduler-level `--dependency` chaining is a
possible future enhancement.

**Why it's convenient**

- One anchor (`pipeline_dir`) namespaces a whole experiment; stages compose
  purely through interpolation, no manual `save_dir`/path plumbing between calls.
- Stages are *append-only*: add a new stage to the config and run it later
  against an existing `pipeline_dir` — it lands beside the earlier stages and
  can read their artifacts via `${pipeline_dir}/<stage>/...`.
- Sweeping over `pipeline_dir` fans one stage out across experiments; sweeping
  over `params.*` fans one experiment across hyperparameters.
- The result tree *is* the index: `results/<xp>/<stage>/` is browsable, and
  each stage dir carries its own `run.lock` / `run.complete` / `results.json`.

**Scope — use this when**

- Stages form a DAG rooted at one directory and mostly share a `params` block.
- You want past experiments to stay browsable and re-runnable stage-by-stage.
- Downstream stages consume upstream *artifacts* (checkpoints, stats files) by
  path rather than needing the upstream *config* rebuilt.

**Limitations & mitigations**

1. **Config drift (the main sharp edge).** A downstream stage rebuilds its
   config from the *current* `pipeline.py`. If you changed an architecture knob
   (`encoder_size`, `img_size`, channel count) since a past experiment trained,
   running e.g. `linear_probe -o pipeline_dir=<old xp>` builds an encoder that
   no longer matches the frozen checkpoint → load failure. The `pipeline_dir`
   anchor only pins *where* files go, not *what* the upstream config was.
   Mitigations, in order of robustness:
   - **Pull frozen values from the upstream `run.lock`** instead of rebuilding
     them, using the `${run_lock:}` resolver:
     ```python
     encoder=py2cfg(
         Encoder,
         size="${run_lock:${pipeline_dir}/train,config.train.encoder.size}",
         img_size="${run_lock:${pipeline_dir}/train,config.train.encoder.img_size}",
     )
     ```
     Now the probe's encoder is defined by whatever trained, not by today's
     defaults. Keep architecture-defining knobs in `params` so there's a single
     dotted key to read back.
   - **Run from the frozen pipeline snapshot.** `proj.save_snapshot(pipeline_dir)`
     writes `pipeline.yaml`; re-run a later stage against the *frozen* config
     with `flexlock-run -c results/myxp/pipeline.yaml -s linear_probe` instead of
     `-d ...pipeline_cfg`. This ignores current `pipeline.py` entirely.
   - Treat architecture-changing edits as a *new* experiment root rather than
     reusing an old `pipeline_dir`.

2. **`smart_run` and code changes.** Caching is safe here: the fingerprint
   includes the git tree hash, so "same config text, different code" is a cache
   *miss*, not a wrong reuse. But that also means a downstream stage won't
   cache-hit a past run whose code has since changed — expected, and the reason
   config drift (above) must be handled explicitly, not via the cache.

3. **Anchor vs `search_dirs` for cross-experiment cache hits.** The sweep form
   runs the *same* stage into *different* `pipeline_dir`s. If you also want a
   fingerprint-index cache hit across experiments, the index root and
   `search_dirs` must span all of them (see §8 / `FLEXLOCK_INDEX`). Within a
   single `pipeline_dir` this is automatic.

4. **`pipeline_dir="???"` is mandatory.** OmegaConf raises if it's left
   unresolved, which is the intended guard — every run must name its experiment.

---

## 15. FAQ / gotchas

**Why must I use `if __name__ == "__main__":` for sweeps?**
`n_jobs > 1` and `isolated=True` use `multiprocessing.spawn` (chosen to
avoid CUDA-fork hazards). Spawn re-imports your launching script in
every worker — without the guard, the worker re-runs `proj.submit(...)`
recursively. Notebooks/REPL are fine.

**Why is `save_dir` resolved eagerly?**
Resolvers like `${vinc:}` inspect the filesystem. If `cfg.save_dir`
were re-resolved at each access, the snapshot phase and the
complete-marker phase would land in different dirs (each call would
advance the version counter). FlexLock resolves it once at submit time
and freezes the string.

**`results.json` is empty after my sweep — what happened?**
The user function returned `None` or a non-dict value. `results.json`
gets `{"result": <value>}` when the return value isn't already a dict.
The driver still loads it back into `ExecutionResult.result`.

**`_target_` fails to import on workers — why?**
Workers re-import the target's module from scratch. If the target lives
in `__main__` (e.g. defined in a notebook or in a script run as
`python script.py`), the worker can't find it. Move the function to a
module and re-run.

**My relative ref `${.x}` got frozen to a literal string — why?**
That happens if you pull the node through `OmegaConf.to_container(...,
resolve=True)` and rebuild it. `proj.get()` is designed to preserve
relative refs; manual round-trips are not.

**`smart_run` keeps missing my cache — why?**
Most common causes: (1) a tracked file in the repo is dirty or
untracked-but-not-ignored, bumping the git tree hash. (2) A data path
changed. (3) The candidate run never wrote `run.complete` (interrupted).
Run `flexlock-diff cur_dir cached_dir` to see exactly what differs.

**I used `${main.save_dir}` and sweep overrides don't propagate.**
Cross-tree refs (`${main.save_dir}`, `${root_anchor}`) are frozen to
their concrete value at `proj.get()` time. Use intra-node refs
(`${.save_dir}`, `${..save_dir}`) for anything that should follow
per-item sweep overrides.

**`flexlock-run -d myproject/defaults.py` works, but
`-d myproject.defaults` says it can't find `defaults`.**
For dotted form, the *last* segment is the variable name, not the
file. `myproject.defaults` means `from myproject import defaults`. If
your layout is `myproject/defaults.py` with `defaults = {...}` inside,
use `myproject.defaults.defaults` or `myproject/defaults.py`.

---

## See also

- [CLI Reference](./cli_reference.md)
- [Python API](./python_api.md)
- [HPC Integration](./hpc_integration.md)
- [Resolvers](./resolvers.md)
- [Debugging](./debugging.md)
- [Reference (env vars + exceptions)](./reference.md)