ZenDag Documentation

Welcome to the official documentation for ZenDag!

ZenDag is a Python framework designed to streamline Machine Learning experimentation workflows by integrating Hydra, DVC, and MLflow.

Getting Started:

• ZenDag Quickstart: Your First Pipeline
  • Prerequisites
  • Step 1: Write Your Python Stage Function
  • Step 2: Define Function Call as Configuration (Hydra-Zen)
  • Step 3: Select Configs & Configure Pipeline (configure.py)
  • Step 4: Run the Pipeline with DVC
  • Data Versioning in Action
  • Conclusion
• Automatic DAGs & DVC Caching with ZenDag
  • Building a Multi-Stage Pipeline
  • Updated configure.py
  • Running configure and Inspecting the DAG
  • Running the Pipeline & DVC Caching
  • Conclusion
• Config Composition & Reusable Components with ZenDag
  • Example: A Reusable File Logger
  • Conclusion
• Sharing Data & Models: DVC Versioning with Intake Catalogs (using fsspec)
  • Recap: DVC for Versioning Artifacts
  • Step 1: Push DVC Data to a Remote
  • Step 2: Tag a Specific Version in Git
  • Step 3: Install Intake and fsspec DVC support
  • Step 4: Creating an Intake Catalog (catalog.yaml) with fsspec
  • Step 5: Using the Intake Catalog (fsspec version)
  • Benefits of the fsspec Approach