FlexLock Usage Guide

A practical, end-to-end tour of the library. Each section answers one question: what does this give me, and how do I get it?

Table of Contents

1. Overview

2. Installation & first run

3. Configuration model

4. Submitting work — three flavors

5. py2cfg — build configs from Python

6. Sweeps

7. Snapshots & reproducibility

8. Smart run / caching

9. HPC backends

10. MLflow integration

11. Debugging

12. Exceptions

13. Environment variables

14. Common recipes

15. FAQ / gotchas

---

1. Overview

FlexLock is a thin orchestration layer around OmegaConf and Python’s multiprocessing / Slurm / PBS. It runs the same function you’d write anyway, and gives you three things in return:

1. A reproducible save_dir. Every submission resolves save_dir eagerly (so resolvers like ${vinc:} advance exactly once) and writes run.lock before the function runs.

2. A snapshot of the environment. Git tree hash for tracked repos, xxh64 of tracked data files, the resolved config — all in run.lock.

3. A cache. When you submit the same fingerprint again, FlexLock returns the cached result instead of re-executing.

It sits between Hydra (config composition) and a job scheduler (Slurm, PBS, local multiprocessing): you write the function, FlexLock handles config merging, save-dir lifecycle, snapshotting, parallel dispatch, and cache lookup.

---

2. Installation & first run

pip install flexlock
# or
conda install -c quentinf00 flexlock

Minimal “hello world”:

# train.py
from flexlock import flexcli

@flexcli
def train(lr: float = 0.01, epochs: int = 10, save_dir: str = 'outputs/train'):
    """Train a model."""
    print(f"lr={lr}, epochs={epochs}, save_dir={save_dir}")
    return {"accuracy": 0.95}

if __name__ == "__main__":
    train()

python train.py -o lr=0.1

After the run, outputs/train/ contains:

outputs/train/
├── run.lock         # config + git tree hash + data hashes, written before exec
├── run.complete     # written after the function returns; cache-hit marker
├── results.json     # the function's return value (if dict-shaped)
└── snapshot/        # filtered git tree if you snapshot repos / data

run.lock is the canonical record. run.complete is what smart_run looks for to decide a run is a real cache hit (a run with only run.lock was interrupted).

---

3. Configuration model

FlexLock configs are OmegaConf DictConfigs. A config that should run something carries a _target_ key whose value is a dotted import path:

# config.yaml
_target_: myproject.train.train
lr: 0.01
epochs: 100
save_dir: outputs/train

At submit time, FlexLock recursively instantiates any nested dict that has _target_, then calls the top-level target with the remaining keys as keyword arguments.

Defaults files

A “defaults” file exports a dict (or DictConfig) under a named variable. Three forms are accepted by -d and Project(defaults=...):

# Dotted module path — variable is the last segment
flexlock-run -d myproject.config.defaults

# File path with explicit variable
flexlock-run -d configs/defaults.py:defaults

# Bare file path — defaults to the `defaults` variable
flexlock-run -d configs/defaults.py

The file-based forms register the loaded file in sys.modules under its file stem, so any _target_ strings captured by py2cfg inside the file remain importable when workers re-import them later. flexlock-run also inserts the current working directory into sys.path, so dotted imports of project-root packages work without PYTHONPATH=..

Overrides and selection

The merge order for flexlock-run is:

defaults  +  -c yaml  +  -m merge_file  +  -o root.overrides
           → -s select_node
           +  -M post-select merge  +  -O post-select overrides
           → execute / sweep

-O and -M apply after selection, so you can override the selected node without disturbing the root.

Swapping a subtree: key=@name

A value starting with @ names a config instead of giving one, and replaces the node (the way Hydra config groups do), rather than merging into it:

flexlock-run -d my.xps.train_x -s main -o main.lit_module.model=@big_model
flexlock-run -d my.xps.train_x -s main -O lit_module.model=@big_model   # after select
flexlock-run -d my.xps.train_x -s main -o main.model=@other.module.huge_model
flexlock-run -d my.xps.train_x -s main --sweep "@small_model,@big_model" \
    --sweep-target main.lit_module.model

• @name is an attribute of the -d module (my.xps above), so building blocks defined next to your presets (small_model = py2cfg(...)) can be swapped by name. @pkg.module.attr reaches a config defined elsewhere.

• Replacement matters: merging big_model into small_model would keep the keys only small_model has. With @, the node is exactly big_model.

• Overrides apply in order, so -o main.model=@big_model main.model.width=16 swaps, then tweaks the new model.

• @@x is the literal string @x. An unknown name is an error listing the module’s config attributes.

• The override is recorded as typed in _preset_; the expanded subtree is in config, so provenance is complete. The same works from Python: Project("my.xps.train_x").submit("main", overrides={"model": "@big_model"}) and in sweep items ({model: "@big_model"} in a sweep file).

This replaces registry resolvers such as ${sel_model:small} over a _MODELS dict.

Relative interpolations

Within a node, prefer relative refs over root-scope refs. They’re the only kind of cross-key reference that survives proj.get() and follows per-item sweep overrides:

# myproject/configs.py
from flexlock import py2cfg
from myproject.train import train

defaults = dict(
    train=py2cfg(
        train,
        save_dir='outputs/train',
        log_dir='${.save_dir}/logs',         # sibling key — relative
        ckpt_dir='${.save_dir}/checkpoints', # sibling key — relative
    ),
)

${.x} resolves against the current node, ${..x} against its parent, ${...x} against its grandparent. These are preserved by proj.get() and by sweep merging, so per-item save_dir overrides correctly propagate into log_dir and ckpt_dir. Root-scope refs (${root_anchor}, ${main.save_dir}) are frozen to their concrete value at proj.get() time, which keeps the returned sub-tree pickleable for HPC dispatch.

save_dir_policy — collision guard (replaces ${vinc:} / ${now:})

Keep save_dir a stable path in configs; save_dir_policy decides what happens when it’s already occupied (default "raise"; "increment" versions it the way ${vinc:} used to, "timestamp" replaces ${now:}). See resolvers.md for the full policy table and migration guide. ${vinc:}/${now:} still work for one deprecation cycle but emit a DeprecationWarning.

save_dir='outputs/train/run'
proj.submit(cfg, save_dir_policy="increment")
# → outputs/train/run_0000, run_0001, ...

Other resolvers: ${latest:glob_pattern} (newest match), ${run_lock:<run_dir>,<dotted.key>} (read a value from an upstream run.lock).

---

4. Submitting work — three flavors

4a. CLI via @flexcli + flexlock-run

For scripts you’ll mostly run from the shell:

# train.py
from flexlock import flexcli

@flexcli
def train(lr: float = 0.01, save_dir: str = 'outputs/train'):
    """Train a model."""
    return {"accuracy": 0.95}

if __name__ == "__main__":
    train()

python train.py -o lr=0.1
flexlock-run -d train.train -O lr=0.1     # equivalent
flexlock-run -d train.train --note "baseline before lr sweep"   # record intent
flexlock-run --help                        # prints argparse help + compiled config

You get an ExecutionResult.result back (the dict), run.lock, run.complete, and results.json on disk.

Pass --note (CLI) or note="…" (proj.submit(cfg, note=…)) to record a free-text intent in run.lock. It never affects caching and shows up in flexlock show/graph/report.

4b. Python Project().submit(cfg)

For multi-stage scripts and notebooks where you want pipeline plumbing (proj.get, proj.defaults, proj.run_stage, …):

from flexlock import Project

proj = Project(defaults='myproject.configs.defaults')

cfg = proj.get('train')
cfg.lr = 0.1
result = proj.submit(cfg)

print(result.save_dir)         # 'outputs/train'
print(result.status)           # 'SUCCESS' or 'CACHED'
print(result['accuracy'])      # dict-like access to result.result

proj.get(key) returns a self-contained sub-tree: root-scope refs are frozen, resolver calls and intra-node refs are preserved.

4c. from flexlock import submit — one-shot

For when you already have a DictConfig and don’t need a Project:

from flexlock import submit, py2cfg
from myproject.train import train

cfg = py2cfg(train, lr=0.01, save_dir='outputs/train')
result = submit(cfg, slurm_config='configs/slurm_gpu.yaml')

submit is Project().submit with all the same kwargs (sweep, smart_run, overrides, force, isolated, …). Use it for ad-hoc one-offs and library code that takes a config and runs it.

---

5. py2cfg — build configs from Python

py2cfg reads a function/class signature and produces a config dict keyed by _target_:

from flexlock import py2cfg
from myproject.train import train
from myproject.models import Transformer

cfg = py2cfg(
    train,
    lr=0.01,
    model=py2cfg(Transformer, layers=12, heads=16),
    save_dir='outputs/train',
)
# {
#   '_target_': 'myproject.train.train',
#   'lr': 0.01,
#   'epochs': 10,                       # captured from signature default
#   'model': {'_target_': 'myproject.models.Transformer', 'layers': 12, 'heads': 16},
#   'save_dir': 'outputs/train',
# }

Use it when:

• you want to refactor a function and have its config follow (signature changes propagate automatically),

• you want nested instantiation (model=py2cfg(...)),

• you want IDE autocomplete and type-checking on the keyword names.

Prefer YAML when configs are written by non-Python users, or when you want a declarative record that lives outside the codebase.

Gotchas:

• The target must be re-importable on workers. py2cfg(train) inside a notebook produces _target_: __main__.train, which won’t import in a spawned subprocess. Move definitions to a module.

• Every default in the signature ends up in the config, including None defaults. Drop them via overrides if you don’t want them in run.lock.

• Decorated functions are unwrapped automatically (py2cfg looks at _original_fn).

---

6. Sweeps

A sweep is a list of override dicts merged into the base config one at a time. Sources:

proj.submit(cfg, sweep=[{'lr': 0.001}, {'lr': 0.01}])              # inline
proj.submit(cfg, sweep_file='configs/grid.yaml')                    # not directly — see below
flexlock-run -d ... --sweep-file configs/grid.yaml
flexlock-run -d ... --sweep "0.001,0.01,0.1" --sweep-target lr
flexlock-run -d ... --sweep-key param_grid                          # list lives in defaults

From Python, use load_sweep to read a file:

from flexlock import load_sweep, Project

proj = Project(defaults='myproject.configs.defaults')
sweep = load_sweep(sweep_file='configs/grid.yaml')

if __name__ == "__main__":
    results = proj.submit('train', sweep=sweep, n_jobs=4)
    for r in results:
        print(r.save_dir, r['accuracy'])

sweep_target

Without sweep_target, each item is merged at the root of the base config. With sweep_target='optimizer.lr', scalar items are placed at that dotted path.

sweep_dir_suffix

When True, each item’s save_dir becomes <base.save_dir>/sweep_{i:04d}/ — items are nested under the sweep root, not siblings to it. This keeps the tasks DB and lineage markers inside the same tree:

proj.submit(cfg, sweep=sweep, sweep_dir_suffix=True)
# outputs/train/sweep_0000/
# outputs/train/sweep_0001/
# outputs/train/run.lock.tasks.db

Containment is validated up-front: if a sweep item’s save_dir falls outside the sweep root, you get a FlexLockValidationError listing the offenders before any work is queued.

Enqueue a sweep for later (--sweep + --enqueue)

--enqueue queue.yaml normally appends the single compiled config to a queue file. Combined with a sweep source it appends one entry per item, each already merged at --sweep-target and frozen (deferred resolvers preserved):

flexlock-run -d ... --sweep "0.001,0.01,0.1" --sweep-target lr --enqueue q.yaml
# q.yaml now has 3 entries; run them later (locally or on HPC) with:
flexlock-run -d ... --sweep-file q.yaml --n_jobs 3

This is the single-stage counterpart of the multi-stage --enqueue batching in §14g — there each item is a composite {_stages_: [...]} pipeline task instead of one plain config.

Per-item interpolations

Because relative refs (${.x}, ${..x}) survive node selection and sweep merging, a sweep that overrides save_dir automatically updates sibling keys that point to it:

from flexlock import py2cfg, Project
from myproject.train import train

cfg = py2cfg(
    train,
    save_dir='outputs/train',
    log_dir='${.save_dir}/logs',          # follows per-item save_dir
    ckpt_dir='${.save_dir}/ckpts',
)

if __name__ == "__main__":
    proj = Project()
    proj.submit(cfg, sweep=[
        {'save_dir': 'outputs/sweep/a'},  # log_dir → outputs/sweep/a/logs
        {'save_dir': 'outputs/sweep/b'},  # log_dir → outputs/sweep/b/logs
    ], n_jobs=2)

n_jobs, isolated, and the __main__ guard

• n_jobs=1 (default, no isolated): runs in-process. No guard needed.

• n_jobs > 1 or isolated=True: uses multiprocessing with the spawn start method (avoids CUDA fork hazards). Spawn re-imports the launching script in each child, so any module-scope proj.submit(...) call must sit under if __name__ == "__main__": or the children will recursively re-launch.

• Notebooks and REPL contexts are fine.

After a parallel or isolated=True run, workers write results.json into each per-task save_dir. The driver loads them back and populates ExecutionResult.result, so the return value is available downstream just like in serial execution.

---

7. Snapshots & reproducibility

Every submit with a save_dir writes run.lock with three sections:

# outputs/train/run.lock
timestamp: "2026-06-01T14:30:00"
repos:
  myproject:
    commit: "abc123…"
    tree: "def456…"               # this is what smart_run compares
    is_dirty: false
    path: "/abs/path/to/repo"
data:
  train_data: "xxh64_789xyz…"
config:
  _target_: myproject.train.train
  lr: 0.01
  save_dir: outputs/train

Declaring what to track

from flexlock import py2cfg
from myproject.train import train

cfg = py2cfg(
    train,
    input_path='data/train.csv',
    save_dir='outputs/train',
    _snapshot_=dict(
        repos={
            'myproject': '.',                       # string shorthand → {'path': '.'}
            'mylib': {'path': 'libs/mylib'},        # explicit
            'pkg':   {'module': 'pkg'},             # resolved via importlib
            'filtered': {                           # filtered comparison
                'path': '.',
                'include': ['src/model/**'],
                'exclude': ['tests/**'],
            },
        },
        data={'train': '${...input_path}'},
        prevs=['${...upstream_run_dir}'],           # link to upstream runs
    ),
)

Relative refs inside _snapshot_ need three dots. They resolve from their own position in the tree, and data values and prevs items both sit two levels below the stage node (stage._snapshot_.data.train), so climbing back to a sibling of _snapshot_ takes ${...key}: data.train → data → _snapshot_ → stage. A single-dot ${.input_path} would look for input_path inside the data mapping and raise InterpolationKeyError.

When a config has _target_, FlexLock auto-tracks the git repo of the target’s source file — you don’t need to spell repos for the main codebase. Add explicit entries for sibling libraries, vendored dependencies, or filtered comparisons.

snapshot() directly

You can also write a snapshot outside the submit flow:

from flexlock import snapshot

snapshot(cfg, repos={'main': '.'}, data={'train': 'data/train.csv'})

Re-running from a snapshot

run.lock is YAML — pull the config block out and resubmit it:

from omegaconf import OmegaConf
from flexlock import submit

cfg = OmegaConf.load('outputs/train/run.lock').config
submit(cfg, force=True)   # re-execute, overwriting in place

force=True invalidates run.complete (so smart_run doesn’t short- circuit), preserves outputs and run.lock, and re-runs the function.

Presets: which config produced a run

A preset is what you pass on the command line: the -d target (a Python attribute holding a config) plus the -s key. You don’t register anything: every run made through flexlock-run -d/-s, or through Project("pkg.module.attr").get(key), records its preset in the config as a reserved _preset_ key, along with the overrides you typed:

# run.lock
config:
  _target_: sst_ml_mapping.starter.train.train
  _preset_:
    defaults: sst_ml_mapping.starter.xps_glob:train_small_cloud_gap_compact
    select: main
    overrides: ["main.lit_module.lr=1e-4"]   # -o, as typed
    overrides_after_select: []               # -O
    merges: []                               # -m / -M
  ...

_preset_ is provenance only: it never enters the fingerprint (so caching is unaffected), RunDiff ignores it, and instantiate strips it before calling your function. Only runnable nodes (_target_ + save_dir) get one.

When a run completes, FlexLock adds a symlink under the project’s .flexlock/presets/<defaults>/<select>/, which makes the runs of a preset cheap to list:

flexlock runs xps_glob.train_small_cloud_gap_compact       # newest first
flexlock runs train_small_cloud_gap_compact -s main --strict  # no overrides
flexlock presets sst_ml_mapping.starter.xps_glob             # the catalogue
flexlock presets sst_ml_mapping.starter.xps_glob --format md > docs/presets.md

flexlock presets lists every attribute of a config module that holds a runnable stage (at its top level or one level down; deeper nodes such as models or callbacks are building blocks), with the comment block above its definition and its number of runs. It can replace a hand-maintained index of commands. Addresses match by suffix: xps_glob.train_x, xps_glob:train_x and train_x all find pkg.starter.xps_glob:train_x.

Put a .flexlock/ directory at the project root (the project template does) so one set of links serves every results directory; without it the links live next to each results directory’s index, which flexlock runs also finds from the project root. flexlock reindex <results_root> rebuilds the links. Runs made before presets were recorded have no _preset_ and don’t appear.

Referencing another preset’s runs: ${run:}

To use the output of another experiment, reference its preset instead of copying a versioned path:

infer_cloud_gap = dict(
    train_dir="${run:xps_glob.train_small_cloud_gap,main}",   # newest complete run
    main=py2cfg(infer_fn, xp_path="${train_dir}", ...),
)

Add a pin (${run:xps_glob.train_x,main,0005}) to use a specific version, or strict (${run:xps_glob.train_x,main,strict}) to ignore runs made with overrides. The reference is resolved when you submit: the chosen path is logged, frozen into the config (and into --dump/--enqueue output), and added to the run’s lineage. Keep quick tests from being picked up by making them their own preset (train_x_debug = ...). Full reference: resolvers.

---

8. Smart run / caching

proj.submit(cfg, smart_run=True, search_dirs=['outputs/train/'])

Mechanism:

1. Compute a stable fingerprint from cfg (per-repo git tree hashes restricted to include/exclude, data hashes, and the resolved config with save_dir prefix-normalized so output location doesn’t affect it). This uses git write-tree only — no commits or refs are created.

2. Look the fingerprint up in the project-wide index (one indexed SELECT). The index maps a fingerprint to where a completed run lives — a run.lock directory or a sweep task (task_db, task_id), so sweep items are first-class cache entries.

3. On a hit, verify the pointed-to run still exists and is complete (run.complete, or the task’s results.json). A stale pointer self-prunes and is treated as a miss. Only status='done' runs are ever served.

4. On an index miss, fall back to the legacy **/run.lock glob scan + RunDiff (controlled by FLEXLOCK_INDEX_FALLBACK, default on) and backfill the index on a hit, so the slow path self-eliminates.

Rebuild the index at any time with flexlock reindex <results_root>; deleting it is always safe.

The index lives in the nearest .flexlock/ directory above the run (walking up from its parent), or <parent>/.flexlock/ if there is none; $FLEXLOCK_INDEX overrides both. A .flexlock/ at the project root therefore collects every run in the project, which is what flexlock runs and cache lookups from the project root expect. The same directory holds the preset links (§7).

When caches miss

• The git tree hash changed (any tracked file modified, including untracked-but-not-ignored files).

• A tracked data path’s xxh64 changed.

• The config changed (after resolution, ignoring _snapshot_).

• The candidate run never wrote run.complete.

To debug a miss:

flexlock-diff dirs outputs/train/run_0001 outputs/train/run_0002
# exit code: 0 = match, 1 = differ, 2 = error; add --format json for scripting
flexlock why outputs/train/run_0001 outputs/train/run_0002   # + the commits between them

Failure records (run.error)

When a run’s user function raises, FlexLock writes a run.error JSON sidecar next to run.lock (exc_type, exc_message, full traceback, timestamp; plus task_id/node for sweep tasks). The original exception still propagates unwrapped — capture never masks it. Triage without opening sqlite:

flexlock show outputs/train/run_0001 --format json    # .status == "failed", .error.*

A successful re-run (force=True / --force) clears the stale run.error. See Agentic Workflows for the full schema and triage guidance.

Knobs

Knob	Effect
smart_run=False	Always execute
force=True	Invalidate run.complete (and, for sweeps, per-item markers + the task DB), then execute
FLEXLOCK_NO_CACHE=1	Disable the on-disk data-hash cache
FLEXLOCK_INDEX=/path/index.db	Explicit fingerprint-index location
FLEXLOCK_INDEX_FALLBACK=0	Skip the legacy glob scan on an index miss
match_include / match_exclude	Override git path filters at compare time
search_dirs=[...]	Where to look for cached runs

Combine with sweeps: re-running a 32-job sweep with smart_run=True re-executes only the ones whose fingerprints don’t match an existing completed run.

---

9. HPC backends

Slurm and PBS are supported via YAML config files. The same path works for submit (single job) and submit(..., sweep=..., n_jobs=N) (array job).

# configs/slurm_gpu.yaml
startup_lines:
  - "#SBATCH --job-name=flexlock"
  - "#SBATCH --gres=gpu:1"
  - "#SBATCH --cpus-per-task=8"
  - "#SBATCH --mem=32G"
  - "#SBATCH --time=04:00:00"
  - "#SBATCH --array=0-31"
  - "module load cuda/11.8"
  - "source activate myenv"
python_exe: "python"

proj.submit(cfg, sweep=sweep, slurm_config='configs/slurm_gpu.yaml')

# configs/pbs_gpu.yaml — container example
startup_lines:
  - "#PBS -l select=1:ncpus=8:mem=32gb"
  - "#PBS -l walltime=04:00:00"
  - "#PBS -J 0-31"
  - "cd $PBS_O_WORKDIR"
python_exe: |
  singularity run --nv
  --bind $(pwd):/workspace --pwd /workspace
  env.sif python

wait=False submits and returns immediately. wait=True blocks until the array finishes; the default timeout comes from FLEXLOCK_DEFAULT_TIMEOUT (seconds).

Preview the rendered submission script without submitting:

proj.submit(cfg, slurm_config='configs/slurm_gpu.yaml', dry_run=True)
# Prints the generated Slurm script plus any validation warnings.

Multi-stage pipelines on HPC

A multi-stage selection runs on HPC too: array workers fan out across sweep items, and each item’s stages run in order on the worker that claims it.

flexlock-run -d project.pipeline.pipeline_cfg -s train linear_probe \
    --sweep-target pipeline_dir --sweep results/xp1,results/xp2 \
    --slurm-config configs/slurm_gpu.yaml

From Python this is Project.submit_pipeline(items, slurm_config=...), where items[i] is the list of frozen stage configs for item i. Because all stages of an item share one scheduler job, size the job for the heaviest stage (see the one-job-per-item limitation in §14g). dry_run=True renders the script without queueing anything.

Anything that goes wrong on the backend side — bad YAML, queue rejection, task-DB lock contention — is raised as FlexLockBackendError.

---

10. MLflow integration

mlflow_context wraps an MLflow run that’s bound to a physical save_dir. It implements an “always new + deprecate old” strategy so re-runs into the same directory don’t pollute the previous MLflow run:

from flexlock import flexcli, mlflow_context
import mlflow

@flexcli
def train(lr: float = 0.01, save_dir: str = 'outputs/train'):
    with mlflow_context(save_dir, experiment_name='my-exp') as run:
        mlflow.log_metric('train_loss', 0.32)
        mlflow.log_metric('val_accuracy', 0.95)
    return {"accuracy": 0.95}

if __name__ == "__main__":
    train()

The context inherits user tags from the previously active run for the same directory, so pipeline tags like pipeline_run=run_001 persist across stages without manual plumbing. MLflow not installed? The context is a no-op and yields None.

---

11. Debugging

@debug_on_fail

from flexlock import debug_on_fail

@debug_on_fail
def train(...):
    ...

On exception, the decorator either drops you into PDB (script context) or injects the function’s locals into the caller’s scope (notebook context). Strategy is chosen by FLEXLOCK_DEBUG_STRATEGY:

Value	Behavior
auto (default)	PDB for scripts, locals injection for notebooks
pdb	Always PDB post-mortem
inject	Always inject locals

Activation

Two control surfaces, two semantics:

• @flexcli / flexlock-run --debug: opt in to wrapping the user function with debug_on_fail. FLEXLOCK_DEBUG=1 (or --debug) enables it.

• Manually applied @debug_on_fail: active by default. To turn it off (e.g. in production), set FLEXLOCK_NODEBUG=1 — the decorator returns the original function unchanged. Note: FLEXLOCK_DEBUG=0 does not disable a manually applied decorator.

The decorator accepts (and warns about) legacy kwargs like stack_depth=, so old snippets won’t blow up — they’re just ignored.

print_config

flexlock-run -d configs/defaults.py -s train -O lr=0.1 --print-config

Prints the fully-merged config plus the target function’s docstring, then exits. From Python:

proj.submit('train', overrides={'lr': 0.1}, print_config=True)
# When sweep is also passed, prints one config per sweep item.

flexlock-run --help prints argparse help followed by the same compiled config view.

Logging to file

from pathlib import Path
from flexlock import flexcli, log_to_file
from loguru import logger

@flexcli
def train(lr: float = 0.01, save_dir: str = 'outputs/train'):
    with log_to_file(Path(save_dir) / 'debug.log'):
        logger.debug(f"starting with lr={lr}")
        ...

log_to_file is a context manager that adds a Loguru sink for its lifetime and removes it on exit. Importable from the package root.

---

12. Exceptions

All FlexLock exceptions inherit from FlexLockError.

Exception	Wraps
FlexLockConfigError	Invalid config structure, bad _target_, malformed _snapshot_
FlexLockValidationError	Sweep save_dir containment, ambiguous sweep source, missing select
FlexLockSnapshotError	Can’t write run.lock, git/data hash failures
FlexLockCacheError	Cache read / fingerprint comparison failures
FlexLockExecutionError	FlexLock-side dispatch failures (NOT user exceptions — see below)
FlexLockBackendError	Slurm/PBS submission failures, queue errors, task DB lock issues

Important: exceptions raised inside the user’s target function are not wrapped. They propagate as the original type (ValueError, RuntimeError, …). Catch them directly:

from flexlock import FlexLockBackendError

try:
    result = proj.submit(cfg, slurm_config='configs/slurm_gpu.yaml')
except FlexLockBackendError as e:
    print(f"HPC failed: {e}")
except ValueError as e:
    print(f"User function raised: {e}")

---

13. Environment variables

Variable	Default	Effect
FLEXLOCK_DEBUG	false	Opt into debug wrapping via @flexcli / flexlock-run
FLEXLOCK_NODEBUG	false	Disable a manually applied @debug_on_fail
FLEXLOCK_DEBUG_STRATEGY	auto	auto / pdb / inject
FLEXLOCK_NO_CACHE	false	Disable on-disk data-hash cache
FLEXLOCK_CACHE	~/.cache	Cache base dir
FLEXLOCK_DEFAULT_N_JOBS	1	Default --n_jobs if not given
FLEXLOCK_DEFAULT_TIMEOUT	3600	HPC wait timeout (seconds)
FLEXLOCK_POLL_INTERVAL	10	HPC poll interval (seconds)
FLEXLOCK_WARN_SMART_RUN	true	Warn when smart_run=True and search_dirs=None
FLEXLOCK_TIMESTAMP_FORMAT	%Y-%m-%dT%H-%M-%S	Directory timestamp format
FLEXLOCK_CONFIGURE_LOGGING	true	Configure Loguru on import

Boolean values accept 1/true/yes/on and 0/false/no/off. See reference.md for the full list.

---

14. Common recipes

14a. 16-job hyperparameter sweep on Slurm

# scripts/sweep.py
from flexlock import Project, load_sweep

if __name__ == "__main__":
    proj = Project(defaults='myproject.configs.defaults')
    sweep = [{'lr': lr, 'batch_size': bs}
             for lr in [0.001, 0.01, 0.1, 1.0]
             for bs in [16, 32, 64, 128]]   # 16 items

    results = proj.submit(
        'train',
        sweep=sweep,
        sweep_dir_suffix=True,
        slurm_config='configs/slurm_gpu.yaml',  # set #SBATCH --array=0-15
        smart_run=True,
    )

    best = max(results, key=lambda r: r.get('val_accuracy', 0))
    print(f"best: {best.save_dir} → {best['val_accuracy']}")

14b. Resume a partially-finished sweep

Re-run the same script. smart_run=True skips every completed item (has both run.lock and run.complete); only interrupted items re-execute. No special command needed.

14c. Re-run only the runs that failed

from pathlib import Path
from flexlock import Project

proj = Project(defaults='myproject.configs.defaults')

# Items that started but didn't complete
incomplete = [d for d in Path('outputs/sweep').glob('sweep_*/')
              if (d / 'run.lock').exists() and not (d / 'run.complete').exists()]

if __name__ == "__main__":
    for d in incomplete:
        cfg = proj.get('train')
        cfg.save_dir = str(d)
        proj.submit(cfg, force=True)   # re-exec into same dir

14d. Pull sweep results back as a DataFrame

import pandas as pd

rows = []
for r in results:        # list[ExecutionResult] from proj.submit(..., sweep=...)
    rows.append({
        'save_dir': r.save_dir,
        'lr': r.cfg.lr,
        'batch_size': r.cfg.batch_size,
        'accuracy': r.get('accuracy'),
    })
df = pd.DataFrame(rows)

14e. Single isolated GPU run

When you want the child’s CUDA context to die at exit (so a later sweep can fork without the parent holding a stale CUDA handle):

from flexlock import submit, py2cfg
from myproject.train import train

cfg = py2cfg(train, lr=0.01, save_dir='outputs/train')

if __name__ == "__main__":
    result = submit(cfg, isolated=True)
    print(result['accuracy'])

14e-bis. Pick a stage to run with fzf

flexlock stages lists a defaults tree’s runnable stages without resolving the config (so ${vinc:} never fires):

DEF=myproject.pipeline.cfg
flexlock-run -d "$DEF" -s "$(flexlock stages -d "$DEF" --format keys | fzf)"

14f. Multi-stage pipeline with auto-discovery

from flexlock import Project

proj = Project(defaults='myproject.configs.defaults')

prep_cfg = proj.get('preprocess')
prep_cfg.save_dir = 'outputs/exp_001/preprocess'
proj.run_stage(prep_cfg)

train_cfg = proj.get('train')
train_cfg.data_dir = prep_cfg.save_dir         # propagated by run_stage
train_cfg.save_dir = 'outputs/exp_001/train'
proj.run_stage(train_cfg, isolated=True)

proj.save_snapshot('outputs/exp_001')          # writes pipeline.yaml

run_stage auto-discovers search_dirs by scanning sibling experiment directories with the same stage name, so re-running with different hyperparameters cache-hits where it can.

---

14g. Experiment-directory pattern (pipeline_dir anchor)

A single config declares every stage of a project and nests them all under one experiment root interpolation anchor. Each stage’s save_dir — and every cross-stage reference (an upstream checkpoint, a stats file) — is written relative to that anchor:

# project/pipeline.py
from flexlock import py2cfg

def make_pipeline_cfg():
    return dict(
        pipeline_dir="???",                       # required: the experiment root

        params=dict(lr=1e-4, encoder_size="small", img_size=128),

        train=py2cfg(
            train_stage,
            save_dir="${pipeline_dir}/train",
            lr="${params.lr}",
            encoder=py2cfg(Encoder, size="${params.encoder_size}",
                           img_size="${params.img_size}"),
        ),
        linear_probe=py2cfg(
            probe_stage,
            save_dir="${pipeline_dir}/linear_probe",
            checkpoint="${pipeline_dir}/train/checkpoints/best.ckpt",   # upstream artifact
            encoder=py2cfg(Encoder, size="${params.encoder_size}",
                           img_size="${params.img_size}"),
        ),
    )

pipeline_cfg = make_pipeline_cfg()

Run one stage into an experiment; the anchor decides where it lands:

# train lands in results/myxp/train
flexlock-run -d project.pipeline.pipeline_cfg -s train -o pipeline_dir=results/myxp

# add a stage later against the SAME experiment -> results/myxp/linear_probe,
# sitting next to the training run and reading its checkpoint
flexlock-run -d project.pipeline.pipeline_cfg -s linear_probe -o pipeline_dir=results/myxp

# run one stage across several experiments at once
flexlock-run -d project.pipeline.pipeline_cfg -s linear_probe \
    --sweep pipeline_dir=results/myxp1,results/myxp2

# run several stages of ONE experiment in order (space- or comma-separated)
flexlock-run -d project.pipeline.pipeline_cfg -s train linear_probe \
    -o pipeline_dir=results/myxp

# run several stages across several experiments at once (sweep × stages)
flexlock-run -d project.pipeline.pipeline_cfg -s train linear_probe \
    --sweep-target pipeline_dir --sweep results/myxp1,results/myxp2 --n_jobs 2

# same, on Slurm — array workers fan out across experiments
flexlock-run -d project.pipeline.pipeline_cfg -s train linear_probe \
    --sweep-target pipeline_dir --sweep results/myxp1,results/myxp2 \
    --slurm-config slurm.yaml

Passing several stages to -s runs them in order as a pipeline. Each sweep item becomes one composite pipeline task: its stages run sequentially on a single worker, so a downstream stage always sees the upstream stage’s artifacts already on disk (that’s the whole ordering contract — stages compose through ${pipeline_dir}/<stage>/... paths, not in-memory wiring). Parallelism and HPC array workers fan out across items, never within one item. With no sweep, that’s a single composite task running locally in-process.

Each sweep item is merged into the root config before stage selection (respecting --sweep-target as a root dot-path), then every stage is re-selected and re-frozen from that merged root — so -s train linear_probe --sweep-target pipeline_dir --sweep xp1,xp2 builds 2 items × 2 stages, and each item’s stages land under its own pipeline_dir. With --check-exists, stages already completed for a pipeline_dir are skipped, so re-running the sequence resumes where it stopped.

Multi-stage composes with --sweep*, --slurm-config/--pbs-config, and --enqueue. It is only incompatible with -O/-M and -e, which target the selected node (ambiguous across a sequence) — pass shared values as root-level overrides (-o pipeline_dir=..., -m, -c) instead.

Batching with --enqueue. -s train linear_probe --enqueue queue.yaml appends one composite {_stages_: [...]} entry per sweep item to the queue file (deferred resolvers like ${run_lock:}/${latest:} are preserved unresolved). Run the whole queue later — including on HPC — with flexlock-run -d ... --sweep-file queue.yaml; composite entries run as-is (never re-merged with the base config), and plain entries in the same file are merged into the compiled node config and run as single-stage pipelines.

Stage records. Each executed stage keeps its own config, fingerprint, lineage, code-drift information, and preset link. By default its directory contains a full run.lock. With FLEXLOCK_TASK_RECORD=db, the shared task row stores snapshots keyed by absolute stage directory; the normal record reader selects the appropriate stage through its .flexlock_marker. Downstream ${run_lock:...} references can read an upstream stage before the whole task finishes. Exporting a composite task includes all stage snapshots with the master’s code and environment information. Compare individual stage directories with flexlock-diff; a composite task ID names multiple records.

One job per item (accepted limitation). All stages of an item run inside one scheduler job / worker slot, so per-stage HPC resource configs (e.g. a CPU extract stage then a GPU train stage with different --gres) are not possible. Give the whole pipeline the resources its heaviest stage needs, or split the stages into separate submissions. Scheduler-level --dependency chaining is a possible future enhancement.

Why it’s convenient

• One anchor (pipeline_dir) namespaces a whole experiment; stages compose purely through interpolation, no manual save_dir/path plumbing between calls.

• Stages are append-only: add a new stage to the config and run it later against an existing pipeline_dir — it lands beside the earlier stages and can read their artifacts via ${pipeline_dir}/<stage>/....

• Sweeping over pipeline_dir fans one stage out across experiments; sweeping over params.* fans one experiment across hyperparameters.

• The result tree is the index: results/<xp>/<stage>/ is browsable, and each stage dir carries its own run.lock / run.complete / results.json.

Scope — use this when

• Stages form a DAG rooted at one directory and mostly share a params block.

• You want past experiments to stay browsable and re-runnable stage-by-stage.

• Downstream stages consume upstream artifacts (checkpoints, stats files) by path rather than needing the upstream config rebuilt.

Limitations & mitigations

1. Config drift (the main sharp edge). A downstream stage rebuilds its config from the current pipeline.py. If you changed an architecture knob (encoder_size, img_size, channel count) since a past experiment trained, running e.g. linear_probe -o pipeline_dir=<old xp> builds an encoder that no longer matches the frozen checkpoint → load failure. The pipeline_dir anchor only pins where files go, not what the upstream config was. Mitigations, in order of robustness:

   • Pull frozen values from the upstream run.lock instead of rebuilding them, using the ${run_lock:} resolver:

     encoder=py2cfg(
         Encoder,
         size="${run_lock:${pipeline_dir}/train,config.train.encoder.size}",
         img_size="${run_lock:${pipeline_dir}/train,config.train.encoder.img_size}",
     )
     

     Now the probe’s encoder is defined by whatever trained, not by today’s defaults. Keep architecture-defining knobs in params so there’s a single dotted key to read back.

   • Run from the frozen pipeline snapshot. proj.save_snapshot(pipeline_dir) writes pipeline.yaml; re-run a later stage against the frozen config with flexlock-run -c results/myxp/pipeline.yaml -s linear_probe instead of -d ...pipeline_cfg. This ignores current pipeline.py entirely.

   • Treat architecture-changing edits as a new experiment root rather than reusing an old pipeline_dir.

2. smart_run and code changes. Caching is safe here: the fingerprint includes the git tree hash, so “same config text, different code” is a cache miss, not a wrong reuse. But that also means a downstream stage won’t cache-hit a past run whose code has since changed — expected, and the reason config drift (above) must be handled explicitly, not via the cache.

3. Anchor vs search_dirs for cross-experiment cache hits. The sweep form runs the same stage into different pipeline_dirs. If you also want a fingerprint-index cache hit across experiments, the index root and search_dirs must span all of them (see §8 / FLEXLOCK_INDEX). Within a single pipeline_dir this is automatic.

4. pipeline_dir="???" is mandatory. OmegaConf raises if it’s left unresolved, which is the intended guard — every run must name its experiment.

---

15. FAQ / gotchas

Why must I use if __name__ == "__main__": for sweeps? n_jobs > 1 and isolated=True use multiprocessing.spawn (chosen to avoid CUDA-fork hazards). Spawn re-imports your launching script in every worker — without the guard, the worker re-runs proj.submit(...) recursively. Notebooks/REPL are fine.

Why is save_dir resolved eagerly? Resolvers like ${vinc:} inspect the filesystem. If cfg.save_dir were re-resolved at each access, the snapshot phase and the complete-marker phase would land in different dirs (each call would advance the version counter). FlexLock resolves it once at submit time and freezes the string.

results.json is empty after my sweep — what happened? The user function returned None or a non-dict value. results.json gets {"result": <value>} when the return value isn’t already a dict. The driver still loads it back into ExecutionResult.result.

_target_ fails to import on workers — why? Workers re-import the target’s module from scratch. If the target lives in __main__ (e.g. defined in a notebook or in a script run as python script.py), the worker can’t find it. Move the function to a module and re-run.

My relative ref ${.x} got frozen to a literal string — why? That happens if you pull the node through OmegaConf.to_container(..., resolve=True) and rebuild it. proj.get() is designed to preserve relative refs; manual round-trips are not.

smart_run keeps missing my cache — why? Most common causes: (1) a tracked file in the repo is dirty or untracked-but-not-ignored, bumping the git tree hash. (2) A data path changed. (3) The candidate run never wrote run.complete (interrupted). Run flexlock-diff cur_dir cached_dir to see exactly what differs.

I used ${main.save_dir} and sweep overrides don’t propagate. Cross-tree refs (${main.save_dir}, ${root_anchor}) are frozen to their concrete value at proj.get() time. Use intra-node refs (${.save_dir}, ${..save_dir}) for anything that should follow per-item sweep overrides.

flexlock-run -d myproject/defaults.py works, but -d myproject.defaults says it can’t find defaults. For dotted form, the last segment is the variable name, not the file. myproject.defaults means from myproject import defaults. If your layout is myproject/defaults.py with defaults = {...} inside, use myproject.defaults.defaults or myproject/defaults.py.

---

See also

• CLI Reference

• Python API

• HPC Integration

• Resolvers

• Debugging

• Reference (env vars + exceptions)