Skip to content

Integrating Experiment Trackers with MCP #3561

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 5 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 15 additions & 4 deletions docs/book/component-guide/experiment-trackers/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,10 +50,21 @@ zenml experiment-tracker flavor list

Every Experiment Tracker has different capabilities and uses a different way of logging information from your pipeline steps, but it generally works as follows:

* first, you have to configure and add an Experiment Tracker to your ZenML stack
* next, you have to explicitly enable the Experiment Tracker for individual steps in your pipeline by decorating them with the included decorator
* in your steps, you have to explicitly log information (e.g. models, metrics, data) to the Experiment Tracker same as you would if you were using the tool independently of ZenML
* finally, you can access the Experiment Tracker UI to browse and visualize the information logged during your pipeline runs. You can use the following code snippet to get the URL of the experiment tracker UI for the experiment linked to a certain step of your pipeline run:
* first, you have to configure and add an Experiment# Integrating Experiment Trackers with MCP

The ZenML Model Control Plane (MCP) is designed to manage and track machine learning models independently, providing features like version control and metadata tracking. However, integrating an experiment tracker can enhance the MCP's capabilities by providing additional logging and visualization features for experiments and model performance.

## Benefits of Integrating Experiment Trackers with MCP

- **Enhanced Logging and Visualization**: Experiment trackers provide extensive UIs that allow users to browse, visualize, and compare experiment data, which complements the MCP's model management capabilities.
- **Popular Experiment Trackers**: ZenML supports integration with popular experiment trackers such as Comet, MLflow, Neptune, and Weights & Biases.

## How to Integrate Experiment Trackers with MCP

1. **Add an Experiment Tracker to your ZenML Stack**: Configure and add an experiment tracker to your ZenML stack.
2. **Enable the Experiment Tracker for Pipeline Steps**: Explicitly enable the experiment tracker for individual steps in your pipeline by decorating them with the included decorator.
3. **Log Information to the Experiment Tracker**: In your steps, log information (e.g., models, metrics, data) to the experiment tracker as you would if using the tool independently of ZenML.
4. **Access the Experiment Tracker UI**: Use the following code snippet to get the URL of the experiment tracker UI for the experiment linked to a certain step of your pipeline run:

```python
from zenml.client import Client
Expand Down
41 changes: 41 additions & 0 deletions docs/book/component-guide/experiment-trackers/mlflow.md
Original file line number Diff line number Diff line change
Expand Up @@ -113,6 +113,47 @@ Read more about [ZenML Secrets](https://docs.zenml.io/how-to/project-setup-and-m

For more, up-to-date information on the MLflow Experiment Tracker implementation and its configuration, you can have a look at [the SDK docs](https://sdkdocs.zenml.io/latest/integration_code_docs/integrations-mlflow.html#zenml.integrations.mlflow) .

## Integrating Experiment Trackers with MCP

### Overview of MCP

The ZenML Model Control Plane (MCP) is designed to manage and track machine learning models independently. It provides features like version control, metadata tracking, and storage of model artifacts, allowing you to register, version, and manage models as first-class citizens in ZenML.

### Enhancing MCP with Experiment Trackers

While the MCP functions independently, integrating experiment trackers can enhance its capabilities by providing additional logging and visualization features for experiments and model performance. Experiment trackers offer extensive UIs that allow users to browse, visualize, and compare experiment data, complementing the MCP's model management features.

### Popular Experiment Trackers

Some popular experiment trackers that can be integrated with ZenML include:

- Comet
- MLflow
- Neptune
- Weights & Biases

### Integration Steps

To integrate an experiment tracker with the MCP, follow these steps:

1. **Configure and Add an Experiment Tracker**: Register the experiment tracker as part of your ZenML stack.

2. **Enable the Experiment Tracker for Pipeline Steps**: Decorate individual steps in your pipeline to enable logging with the experiment tracker.

3. **Log Information**: Explicitly log models, metrics, and data to the experiment tracker within your pipeline steps.

4. **Access the Experiment Tracker UI**: Use the experiment tracker UI to browse and visualize logged information.

Example code snippet for accessing the experiment tracker UI:

```python
from zenml.client import Client

pipeline_run = Client().get_pipeline_run("<PIPELINE_RUN_NAME>")
step = pipeline_run.steps["<STEP_NAME>"]
experiment_tracker_url = step.run_metadata["experiment_tracker_url"].value
```

## How do you use it?

To be able to log information from a ZenML pipeline step using the MLflow Experiment Tracker component in the active stack, you need to enable an experiment tracker using the `@step` decorator. Then use MLflow's logging or auto-logging capabilities as you would normally do, e.g.:
Expand Down
22 changes: 22 additions & 0 deletions docs/book/component-guide/experiment-trackers/vertexai.md
Original file line number Diff line number Diff line change
Expand Up @@ -259,6 +259,28 @@ def tf_trainer(...):
```
{% endhint %}

### Integrating Experiment Trackers with MCP

The ZenML Model Control Plane (MCP) is designed to manage and track machine learning models independently, providing features like version control and metadata tracking. However, integrating experiment trackers can enhance the MCP by providing additional logging and visualization features for experiments and model performance.

#### Benefits of Integration
- **Enhanced Logging and Visualization**: Experiment trackers offer extensive UIs that allow users to browse, visualize, and compare experiment data, complementing the MCP's capabilities.
- **Popular Experiment Trackers**: ZenML supports integration with popular experiment trackers like Comet, MLflow, Neptune, and Weights & Biases.

#### Integration Steps
1. **Configure and Add an Experiment Tracker**: Add an experiment tracker to your ZenML stack.
2. **Enable the Experiment Tracker**: Decorate individual pipeline steps to enable the experiment tracker.
3. **Log Information**: Explicitly log models, metrics, and data to the experiment tracker.
4. **Access the UI**: Use the experiment tracker UI to visualize logged information.

```python
from zenml.client import Client

pipeline_run = Client().get_pipeline_run("<PIPELINE_RUN_NAME>")
step = pipeline_run.steps["<STEP_NAME>"]
experiment_tracker_url = step.run_metadata["experiment_tracker_url"].value
```

### Experiment Tracker UI

You can find the URL of the Vertex AI experiment linked to a specific ZenML run via the metadata of the step in which the experiment tracker was used:
Expand Down
26 changes: 26 additions & 0 deletions docs/book/how-to/popular-integrations/mlflow.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,32 @@
description: Learn how to use the MLflow Experiment Tracker with ZenML.
---

# Integrating Experiment Trackers with MCP

## Overview of MCP

The ZenML Model Control Plane (MCP) is designed to manage and track machine learning models independently, providing features like version control and metadata tracking. It acts as a model registry, allowing you to register, version, and manage models as first-class citizens in ZenML.

## Enhancing MCP with Experiment Trackers

While the MCP functions independently, integrating experiment trackers can enhance its capabilities by providing additional logging and visualization features for experiments and model performance. Experiment trackers offer extensive UIs that allow users to browse, visualize, and compare logged information, augmenting the MCP's functionalities.

## Popular Experiment Trackers

Some popular experiment trackers that can be integrated with ZenML include:
- Comet
- MLflow
- Neptune
- Weights & Biases

## How to Integrate Experiment Trackers with MCP

To integrate an experiment tracker with the MCP, follow these steps:
1. Configure and add an Experiment Tracker to your ZenML stack.
2. Enable the Experiment Tracker for individual steps in your pipeline by decorating them with the included decorator.
3. Log information (e.g., models, metrics, data) to the Experiment Tracker as you would independently.
4. Access the Experiment Tracker UI to browse and visualize the logged information.

# MLflow Experiment Tracker

The ZenML MLflow Experiment Tracker integration and stack component allows you to log and visualize information from your pipeline steps using MLflow, without having to write extra MLflow code.
Expand Down
Loading