diff --git a/docs/book/getting-started/core-concepts.md b/docs/book/getting-started/core-concepts.md index 84f35ae5332..3180f6599f8 100644 --- a/docs/book/getting-started/core-concepts.md +++ b/docs/book/getting-started/core-concepts.md @@ -19,6 +19,43 @@ If you prefer visual learning, this short video demonstrates the key concepts co First, let's look at the main concepts which play a role during the development stage of an ML workflow with ZenML. +### Iterating Over Pipeline Steps Before Execution + +ZenML does not provide a direct API for iterating over steps before execution, so manual management of steps is necessary. This can be useful for scenarios where you need to inspect or modify steps programmatically before running the pipeline. + +Here's an example of how you can manually manage steps in a list and iterate over them before running the pipeline: + +```python +from zenml import pipeline, step + +@step +def step_one() -> str: + return "Step one completed." + +@step +def step_two() -> str: + return "Step two completed." + +@pipeline +def my_pipeline(): + steps = [step_one, step_two] + for step_func in steps: + result = step_func() + print(result) + +# Define the pipeline +pipeline_instance = my_pipeline + +# Loop through the steps before running +for step_func in pipeline_instance.__closure__[0].cell_contents: + print(f"Preparing to run: {step_func.__name__}") + +# Run the pipeline +pipeline_instance() +``` + +This code manually manages the steps in a list and iterates over them before running the pipeline. Potential use cases for this feature include dynamic step modification or pre-execution validation. + #### Step Steps are functions annotated with the `@step` decorator. The easiest one could look like this. diff --git a/docs/book/how-to/pipeline-development/README.md b/docs/book/how-to/pipeline-development/README.md index 47174814293..0dbad12eb0e 100644 --- a/docs/book/how-to/pipeline-development/README.md +++ b/docs/book/how-to/pipeline-development/README.md @@ -4,4 +4,41 @@ icon: paintbrush-pencil # Pipeline Development -This section covers all aspects of pipeline development in ZenML. \ No newline at end of file +This section covers all aspects of pipeline development in ZenML. + +## Iterating Over Pipeline Steps Before Execution + +ZenML does not provide a direct API for iterating over steps before execution, so manual management of steps is necessary. This can be useful for scenarios such as dynamic step modification or pre-execution validation. + +Here's an example of how you can manually manage steps in a list and iterate over them before running the pipeline: + +```python +from zenml import pipeline, step + +@step +def step_one() -> str: + return "Step one completed." + +@step +def step_two() -> str: + return "Step two completed." + +@pipeline +def my_pipeline(): + steps = [step_one, step_two] + for step_func in steps: + result = step_func() + print(result) + +# Define the pipeline +pipeline_instance = my_pipeline + +# Loop through the steps before running +for step_func in pipeline_instance.__closure__[0].cell_contents: + print(f"Preparing to run: {step_func.__name__}") + +# Run the pipeline +pipeline_instance() +``` + +This code manually manages the steps in a list and iterates over them before running the pipeline. Potential use cases for this feature include dynamic step modification or pre-execution validation. \ No newline at end of file diff --git a/docs/book/how-to/pipeline-development/build-pipelines/README.md b/docs/book/how-to/pipeline-development/build-pipelines/README.md index 174e8b1f62f..114963092ce 100644 --- a/docs/book/how-to/pipeline-development/build-pipelines/README.md +++ b/docs/book/how-to/pipeline-development/build-pipelines/README.md @@ -33,6 +33,18 @@ def train_model(data: dict) -> None: def simple_ml_pipeline(): dataset = load_data() train_model(dataset) + +# Iterating Over Pipeline Steps Before Execution + +# ZenML does not provide a direct API for iterating over steps before execution. +# You can manually manage steps in a list and iterate over them before running the pipeline. +# This is useful for scenarios like dynamic step modification or pre-execution validation. + +# Example: +steps = [load_data, train_model] +for step_func in steps: + print(f"Preparing to run: {step_func.__name__}") + ``` You can now run this pipeline by simply calling the function: diff --git a/docs/book/toc.md b/docs/book/toc.md index 1de831647a6..ee0751c0f92 100644 --- a/docs/book/toc.md +++ b/docs/book/toc.md @@ -62,6 +62,7 @@ * [Use failure/success hooks](how-to/pipeline-development/build-pipelines/use-failure-success-hooks.md) * [Fan in, fan out](how-to/pipeline-development/build-pipelines/fan-in-fan-out.md) * [Hyperparameter tuning](how-to/pipeline-development/build-pipelines/hyper-parameter-tuning.md) + * [Iterating Over Pipeline Steps Before Execution](how-to/pipeline-development/build-pipelines/iterating-over-pipeline-steps-before-execution.md) * [Access secrets in a step](how-to/pipeline-development/build-pipelines/access-secrets-in-a-step.md) * [Run an individual step](how-to/pipeline-development/build-pipelines/run-an-individual-step.md) * [Fetching pipelines](how-to/pipeline-development/build-pipelines/fetching-pipelines.md)