{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Intro to Unit Metrics\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To turn complex evidence into actionable insights, you can run a unit metric as a single-value measure to quantify and monitor risks throughout a model's lifecycle.\n",
    "\n",
    "In this interactive notebook, we introduce the concept of _unit metric_ and provide a step-by-step guide on how to define, execute and extract results from these measures. As an example, we use data from a customer churn use case to fit a binary classification model. To illustrate the application of these measures, we show you how to run sklearn classification metrics as unit metrics, demonstrating their utility in quantifying model performance and risk.\n",
    "\n",
    "In Model Risk Management (MRM), the primary objective is to identify, assess, and mitigate the risks associated with the development, implementation, and ongoing use of quantitative models. The process of measuring risk involves the understanding and assessment of evidence generated throw multiple tests across all the model development lifecycle stages, from data collection and data quality to model performance and explainability.\n",
    "\n",
    "**Evidence versus risk**\n",
    "\n",
    "The distinction between evidence and quantifiable risk measures is a critical aspect of MRM. Evidence, in this context, refers to the outputs from various tests conducted throughout the model lifecycle. For instance, a table displaying the number of missing values per feature in a dataset is a form of evidence. It shows where data might be incomplete, which can affect the model's performance and reliability. Similarly, a Receiver Operating Characteristic (ROC) curve is a graphical plot that illustrates the diagnostic ability of a binary classifier system as its discrimination threshold is varied. The curve is evidence of the model's classification performance.\n",
    "\n",
    "However, these pieces of evidence do not offer a direct measure of risk. To quantify risk, one must derive metrics from this evidence that reflect the potential impact on the model's performance and the decisions it informs. For example, the missing data rate, calculated as the percentage of missing values in the dataset, is a quantifiable risk measure that indicates the risk associated with data quality. Similarly, the accuracy score, which measures the proportion of correctly classified labels, acts as an indicator of performance risk in a classification model.\n",
    "\n",
    "**Unit metric**\n",
    "\n",
    "A _Unit metric_ is a single value measure that is used to identify and monitor risks arising from the development of Machine Learning or AI models. This metric simplifies evidence into a single actionable number, that can be monitored and compared over time or across different models or datasets.\n",
    "\n",
    "Properties:\n",
    "\n",
    "- They are the fundamental computation unit that returns a single value.\n",
    "- They quantify risk and can be used to monitor and assess risks associated with a model's entire lifecycle.\n",
    "- Measurable, relevant, and linked to risk areas and critical business processes - e.g., regulatory requirements, risk appetite, model performance, data quality.\n",
    "- Standalone in nature, meaning they do not rely on other metrics for their calculation or interpretation.\n",
    "\n",
    "Incorporating unit metrics into your ML workflow streamlines risk assessment, turning complex analyses into clear, actionable insights.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Contents    \n",
    "- [About ValidMind](#toc1_)    \n",
    "  - [Before you begin](#toc1_1_)    \n",
    "  - [New to ValidMind?](#toc1_2_)    \n",
    "- [Install the ValidMind Library](#toc2_)    \n",
    "- [Initialize the ValidMind Library](#toc3_)    \n",
    "  - [Get your code snippet](#toc3_1_)   \n",
    "- [Load the demo dataset](#toc4_)    \n",
    "- [Train a model for testing](#toc5_)    \n",
    "- [Initialize ValidMind objects](#toc6_)    \n",
    "- [Assign predictions](#toc7_)    \n",
    "- [Running Unit Metrics](#toc8_)    \n",
    "  - [List Available Metrics](#toc8_1_)    \n",
    "  - [Learn More About a Metric](#toc8_2_)    \n",
    "  - [Run ROC AUC Metric](#toc8_3_)    \n",
    "  - [Pass parameters](#toc8_4_)    \n",
    "  - [Other Model Performance Unit Metrics](#toc8_5_)    \n",
    "- [Running the ROC AUC Metric across Multiple Datasets](#toc9_)    \n",
    "  - [Run multiple unit metrics as a single test](#toc9_1_)    \n",
    "- [Comparing Composite Metrics for Multiple Input Combinations](#toc10_)    \n",
    "- [Next steps](#toc11_)    \n",
    "  - [Work with your model documentation](#toc11_1_)    \n",
    "  - [Discover more learning resources](#toc11_2_)    \n",
    "- [Upgrade ValidMind](#toc12_)    \n",
    "\n",
    "<!-- vscode-jupyter-toc-config\n",
    "\tnumbering=false\n",
    "\tanchor=true\n",
    "\tflat=false\n",
    "\tminLevel=2\n",
    "\tmaxLevel=4\n",
    "\t/vscode-jupyter-toc-config -->\n",
    "<!-- THIS CELL WILL BE REPLACED ON TOC UPDATE. DO NOT WRITE YOUR TEXT IN THIS CELL -->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "<a id='toc1_'></a>\n",
    "\n",
    "## About ValidMind \n",
    "\n",
    "ValidMind is a suite of tools for managing model risk, including risk associated with AI and statistical models.\n",
    "\n",
    "You use the ValidMind Library to automate documentation and validation tests, and then use the ValidMind Platform to collaborate on model documentation. Together, these products simplify model risk management, facilitate compliance with regulations and institutional standards, and enhance collaboration between yourself and model validators.\n",
    "\n",
    "<a id='toc1_1_'></a>\n",
    "\n",
    "### Before you begin \n",
    "\n",
    "This notebook assumes you have basic familiarity with Python, including an understanding of how functions work. If you are new to Python, you can still run the notebook but we recommend further familiarizing yourself with the language. \n",
    "\n",
    "If you encounter errors due to missing modules in your Python environment, install the modules with `pip install`, and then re-run the notebook. For more help, refer to [Installing Python Modules](https://docs.python.org/3/installing/index.html).\n",
    "\n",
    "\n",
    "\n",
    "<a id='toc1_2_'></a>\n",
    "\n",
    "### New to ValidMind? \n",
    "\n",
    "If you haven't already seen our documentation on the [ValidMind Library](https://docs.validmind.ai/developer/validmind-library.html), we recommend you begin by exploring the available resources in this section. There, you can learn more about documenting models and running tests, as well as find code samples and our Python Library API reference.\n",
    "\n",
    "<div class=\"alert alert-block alert-info\" style=\"background-color: #B5B5B510; color: black; border: 1px solid #083E44; border-left-width: 5px; box-shadow: 2px 2px 4px rgba(0, 0, 0, 0.2);border-radius: 5px;\"><span style=\"color: #083E44;\"><b>For access to all features available in this notebook, create a free ValidMind account.</b></span>\n",
    "<br></br>\n",
    "Signing up is FREE — <a href=\"https://docs.validmind.ai/guide/configuration/register-with-validmind.html\" style=\"color: #DE257E;\"><b>Register with ValidMind</b></a></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc2_'></a>\n",
    "\n",
    "## Install the ValidMind Library\n",
    "\n",
    "To install the library:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install -q validmind"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc3_'></a>\n",
    "\n",
    "## Initialize the ValidMind Library \n",
    "\n",
    "ValidMind generates a unique _code snippet_ for each registered model to connect with your developer environment. You initialize the ValidMind Library with this code snippet, which ensures that your documentation and tests are uploaded to the correct model when you run the notebook.\n",
    "\n",
    "<a id='toc3_1_'></a>\n",
    "\n",
    "### Get your code snippet\n",
    "\n",
    "1. In a browser, [log in to ValidMind](https://docs.validmind.ai/guide/configuration/log-in-to-validmind.html).\n",
    "\n",
    "2. In the left sidebar, navigate to **Model Inventory** and click **+ Register Model**.\n",
    "\n",
    "3. Enter the model details and click **Continue**. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/register-models-in-inventory.html))\n",
    "\n",
    "   For example, to register a model for use with this notebook, select:\n",
    "\n",
    "   - Documentation template: `Binary classification`\n",
    "   - Use case: `Marketing/Sales - Analytics`\n",
    "\n",
    "   You can fill in other options according to your preference.\n",
    "\n",
    "4. Go to **Getting Started** and click **Copy snippet to clipboard**.\n",
    "\n",
    "Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load your model identifier credentials from an `.env` file\n",
    "\n",
    "%load_ext dotenv\n",
    "%dotenv .env\n",
    "\n",
    "# Or replace with your code snippet\n",
    "\n",
    "import validmind as vm\n",
    "\n",
    "vm.init(\n",
    "    # api_host=\"...\",\n",
    "    # api_key=\"...\",\n",
    "    # api_secret=\"...\",\n",
    "    # model=\"...\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "<a id='toc4_'></a>\n",
    "\n",
    "## Load the demo dataset \n",
    "\n",
    "In this example, we load a demo dataset to fit a customer churn model.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from validmind.datasets.classification import customer_churn as demo_dataset\n",
    "\n",
    "print(\n",
    "    f\"Loaded demo dataset with: \\n\\n\\t• Target column: '{demo_dataset.target_column}' \\n\\t• Class labels: {demo_dataset.class_labels}\"\n",
    ")\n",
    "\n",
    "raw_df = demo_dataset.load_data()\n",
    "raw_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "<a id='toc5_'></a>\n",
    "\n",
    "## Train a model for testing \n",
    "\n",
    "We train a simple customer churn model for our test.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import xgboost as xgb\n",
    "\n",
    "train_df, validation_df, test_df = demo_dataset.preprocess(raw_df)\n",
    "\n",
    "x_train = train_df.drop(demo_dataset.target_column, axis=1)\n",
    "y_train = train_df[demo_dataset.target_column]\n",
    "x_val = validation_df.drop(demo_dataset.target_column, axis=1)\n",
    "y_val = validation_df[demo_dataset.target_column]\n",
    "\n",
    "model = xgb.XGBClassifier(early_stopping_rounds=10)\n",
    "model.set_params(\n",
    "    eval_metric=[\"error\", \"logloss\", \"auc\"],\n",
    ")\n",
    "model.fit(\n",
    "    x_train,\n",
    "    y_train,\n",
    "    eval_set=[(x_val, y_val)],\n",
    "    verbose=False,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "<a id='toc6_'></a>\n",
    "\n",
    "## Initialize ValidMind objects \n",
    "\n",
    "Once the datasets and model are prepared for validation, we initialize ValidMind `dataset` and `model`, specifying features and targets columns. The property `input_id` allows users to uniquely identify each dataset and model. This allows for the creation of multiple versions of datasets and models, enabling us to compute metrics by specifying which versions we want to use as inputs.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import validmind as vm\n",
    "\n",
    "vm_train_ds = vm.init_dataset(\n",
    "    input_id=\"train_dataset\",\n",
    "    dataset=train_df,\n",
    "    target_column=demo_dataset.target_column,\n",
    ")\n",
    "vm_test_ds = vm.init_dataset(\n",
    "    input_id=\"test_dataset\",\n",
    "    dataset=test_df,\n",
    "    target_column=demo_dataset.target_column,\n",
    ")\n",
    "\n",
    "vm_model = vm.init_model(model=model, input_id=\"xbg_model\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "<a id='toc7_'></a>\n",
    "\n",
    "## Assign predictions \n",
    "\n",
    "We can now use the `assign_predictions()` method from the Dataset object to link existing predictions to any model. If no prediction values are passed, the method will compute predictions automatically:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vm_train_ds.assign_predictions(model=vm_model)\n",
    "vm_test_ds.assign_predictions(model=vm_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "<a id='toc8_'></a>\n",
    "\n",
    "## Running Unit Metrics \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "<a id='toc8_1_'></a>\n",
    "\n",
    "### List Available Metrics \n",
    "\n",
    "We can list all available unit metrics by using the `list_metrics()` function from the `unit_metrics` module. This function returns a list of all available metric functions.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from validmind.unit_metrics import list_metrics\n",
    "\n",
    "metrics = list_metrics()\n",
    "metrics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "<a id='toc8_2_'></a>\n",
    "\n",
    "### Learn More About a Metric \n",
    "\n",
    "We can use the `describe_metric()` function to learn more about a metric, specifically its required inputs and optional parameters. Let's check out the details for the ROC AUC metric."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from validmind.unit_metrics import describe_metric\n",
    "\n",
    "describe_metric(\"validmind.unit_metrics.classification.ROC_AUC\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "<a id='toc8_3_'></a>\n",
    "\n",
    "### Run ROC AUC Metric \n",
    "\n",
    "The following snippet shows how to set up and execute a unit metric implementation of the ROC AUC score. We are going to use the `run_metric()` function from the `validmind.unit_metrics` module. This function has a signature very similar to `run_test()` from the `validmind.tests` module, but it is specifically designed to compute unit metrics. It takes the following arguments:\n",
    "\n",
    "- `metric_id`: The unique identifier for the metric, in this case `validmind.unit_metrics.classification.ROC_AUC`\n",
    "- `inputs`: A dictionary containing the input dataset and model or their respective input IDs\n",
    "- `params`: A dictionary containing the keyword arguments for the unit metric (in this case, none are required, but we can pass any `kwargs` that the underlying sklearn implementation accepts)\n",
    "\n",
    "The function returns and displays a result object like a normal ValidMind test, accept that it simply shows the unit metrics value. This result object has a `.log()` method that will log the result to the ValidMind Platform where it can be used for monitoring and model documentation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from validmind.unit_metrics import run_metric\n",
    "\n",
    "result = run_metric(\n",
    "    \"validmind.unit_metrics.classification.ROC_AUC\",\n",
    "    inputs={\n",
    "        \"model\": vm_model,\n",
    "        \"dataset\": vm_test_ds,\n",
    "    },\n",
    ")\n",
    "\n",
    "# Log the result to the ValidMind Platform\n",
    "result.log()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "<a id='toc8_4_'></a>\n",
    "\n",
    "### Pass parameters \n",
    "\n",
    "Unit Metrics can accept parameters as mentioned above. In the case of the ROC AUC and many of the other unit metrics that use an underlying sklearn implementation, these metrics support all parameters of the associated sklearn metric functions. Let's see how we can do this in the case of the above ROC AUC metric.\n",
    "\n",
    "Available parameters for the ROC AUC metric include:\n",
    "\n",
    "- `average`: This parameter is used to compute the average of the ROC AUC score across all classes. The options include `micro`, `macro`, `weighted`, and `samples`.\n",
    "- `sample_weight`: This parameter is used to pass sample weights to the metric.\n",
    "- `max_fpr`: This parameter is used to compute the partial AUC ROC over the range [0, max_fpr].\n",
    "\n",
    "> *See the sklearn documentation for more details on these parameters.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "result = run_metric(\n",
    "    \"validmind.unit_metrics.classification.ROC_AUC\",\n",
    "    inputs={\n",
    "        \"model\": vm_model,\n",
    "        \"dataset\": vm_test_ds,\n",
    "    },\n",
    "    params={\n",
    "        \"average\": \"weighted\",\n",
    "        \"max_fpr\": 1.0,\n",
    "    },\n",
    ")\n",
    "\n",
    "# Log the result to the ValidMind Platform\n",
    "result.log()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "<a id='toc8_5_'></a>\n",
    "\n",
    "### Other Model Performance Unit Metrics \n",
    "\n",
    "In addition to the ROC AUC score, you can compute other classification metrics, such as accuracy score, precision, recall etc. Let's list out all metrics and filter down to classification metrics. Then we can run and log them in a loop."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "metrics = [metric for metric in list_metrics() if \"classification\" in metric]\n",
    "\n",
    "for metric_id in metrics:\n",
    "    describe_metric(metric_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for metric_id in metrics:\n",
    "    result = run_metric(\n",
    "        metric_id,\n",
    "        inputs={\n",
    "            \"model\": vm_model,\n",
    "            \"dataset\": vm_test_ds,\n",
    "        },\n",
    "        show=False, # set show to False to avoid showing an ipywidget output\n",
    "    )\n",
    "    result.log()\n",
    "\n",
    "    # instead of showing the widget, we can print the result value\n",
    "    print(f\"{metric_id}: {result.metric}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "# Comparing Unit Metrics for Different Inputs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "<a id='toc9_'></a>\n",
    "\n",
    "## Running the ROC AUC Metric across Multiple Datasets \n",
    "\n",
    "We can use the `run_test()` function from the `validmind.tests` module and pass in a unit metric ID to run the metric as a test. In practice this is useful for composing and comparing one or more metrics across one or more inptu combinations. If you have followed along some of the other instructional notebooks, you'll remember that `run_test()` accepts an `input_grid` parameter that will allow you to pass multiple values for each input. This will result in the metric being computed for each combination of input values and then combined into a single test result object. For instance, we can compute the ROC AUC score for the train and test datasets of our model. Let's do that now."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from validmind.tests import run_test\n",
    "\n",
    "result = run_test(\n",
    "    \"validmind.unit_metrics.classification.ROC_AUC\",\n",
    "    input_grid={\n",
    "        \"model\": [vm_model],\n",
    "        \"dataset\": [vm_train_ds, vm_test_ds],\n",
    "    },\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "result.log()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "# Composing Test Results from Individual Unit Metrics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "<a id='toc9_1_'></a>\n",
    "\n",
    "### Run multiple unit metrics as a single test \n",
    "\n",
    "Another higher-level way to use unit metrics is to compose multiple metrics into a single test result. This is useful, for instance, if you want to create a single test result for your documentation that contains a table of model performance metrics. In the case of our customer churn model, we can compute the F1, Precision, Recall, Accuracy and ROC AUC scores for the test dataset and combine them into a single test result. Let's do that now."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from validmind.tests import run_test\n",
    "\n",
    "result = run_test(\n",
    "    name=\"Model Performance\",\n",
    "    unit_metrics=[\n",
    "        \"validmind.unit_metrics.classification.F1\",\n",
    "        \"validmind.unit_metrics.classification.Precision\",\n",
    "        \"validmind.unit_metrics.classification.Recall\",\n",
    "        \"validmind.unit_metrics.classification.Accuracy\",\n",
    "        \"validmind.unit_metrics.classification.ROC_AUC\",\n",
    "    ],\n",
    "    inputs={\n",
    "        \"model\": vm_model,\n",
    "        \"dataset\": vm_test_ds,\n",
    "    },\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we take a look at the `result_id` for the result, we'll see that it is a unique identifier that starts with `validmind.composite_metric.<user-supplied-metric-name>`. This will be used to identify this result as coming from a composite metric and is used to rebuild the composite metric as we will see in the next section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "result.result_id"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's go ahead and log the result to save it to the ValidMind Platform."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "result.log()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "# Bringing It All Together"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "<a id='toc10_'></a>\n",
    "\n",
    "## Comparing Composite Metrics for Multiple Input Combinations \n",
    "\n",
    "We can bring all of the above concepts together to create a very sophisticated test result that compares multiple metrics across multiple inputs. This is a very modular and flexible way to create complex test results for your model documentation. Let's go ahead and compare the training and test performance of our model across Accuracy, Precision, Recall, F1 and ROC AUC scores."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "result = run_test(\n",
    "    name=\"Model Performance Comparison Train vs Test\",\n",
    "    unit_metrics=[\n",
    "        \"validmind.unit_metrics.classification.F1\",\n",
    "        \"validmind.unit_metrics.classification.Precision\",\n",
    "        \"validmind.unit_metrics.classification.Recall\",\n",
    "        \"validmind.unit_metrics.classification.Accuracy\",\n",
    "        \"validmind.unit_metrics.classification.ROC_AUC\",\n",
    "    ],\n",
    "    input_grid={\n",
    "        \"model\": [vm_model],\n",
    "        \"dataset\": [vm_train_ds, vm_test_ds],\n",
    "    },\n",
    ")\n",
    "\n",
    "result.log()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "<a id='toc11_'></a>\n",
    "\n",
    "## Next steps \n",
    "\n",
    "You can look at the results of this test suite right in the notebook where you ran the code, as you would expect. But there is a better way — use the ValidMind Platform to work with your model documentation.\n",
    "\n",
    "\n",
    "\n",
    "<a id='toc11_1_'></a>\n",
    "\n",
    "### Work with your model documentation \n",
    "\n",
    "1. From the **Model Inventory** in the ValidMind Platform, go to the model you registered earlier. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/working-with-model-inventory.html))\n",
    "\n",
    "2. Click and expand the **Model Development** section.\n",
    "\n",
    "What you see is the full draft of your model documentation in a more easily consumable version. From here, you can make qualitative edits to model documentation, view guidelines, collaborate with validators, and submit your model documentation for approval when it's ready. [Learn more ...](https://docs.validmind.ai/guide/model-documentation/working-with-model-documentation.html)\n",
    "\n",
    "\n",
    "\n",
    "<a id='toc11_2_'></a>\n",
    "\n",
    "### Discover more learning resources \n",
    "\n",
    "We offer many interactive notebooks to help you document models:\n",
    "\n",
    "- [Run tests & test suites](https://docs.validmind.ai/developer/model-testing/testing-overview.html)\n",
    "- [Code samples](https://docs.validmind.ai/developer/samples-jupyter-notebooks.html)\n",
    "\n",
    "Or, visit our [documentation](https://docs.validmind.ai/) to learn more about ValidMind."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc12_'></a>\n",
    "\n",
    "## Upgrade ValidMind\n",
    "\n",
    "<div class=\"alert alert-block alert-info\" style=\"background-color: #B5B5B510; color: black; border: 1px solid #083E44; border-left-width: 5px; box-shadow: 2px 2px 4px rgba(0, 0, 0, 0.2);border-radius: 5px;\">After installing ValidMind, you’ll want to periodically make sure you are on the latest version to access any new features and other enhancements.</div>\n",
    "\n",
    "Retrieve the information for the currently installed version of ValidMind:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip show validmind"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If the version returned is lower than the version indicated in our [production open-source code](https://github.com/validmind/validmind-library/blob/prod/validmind/__version__.py), restart your notebook and run:\n",
    "\n",
    "```bash\n",
    "%pip install --upgrade validmind\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You may need to restart your kernel after running the upgrade package for changes to be applied."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
