{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Integrate external test providers\n",
    "\n",
    "Register a custom test provider with the ValidMind Library to run your own tests.\n",
    "\n",
    "The ValidMind Library offers the ability to extend the built-in library of tests with custom tests. If you've followed along in the [Implement custom tests notebook](./implement_custom_tests.ipynb), you will be familiar with the process of creating custom tests and running them for use in your model documentation. In that notebook, the tests were defined inline, as a notebook cell, using the `validmind.test` decorator. This works very well when you just want to create one-off, ad-hoc tests or for experimenting. But for more complex, reusable and shareable tests, it is recommended to use a more structured approach.\n",
    "\n",
    "This is where the concept of External Test Providers come in. A test \"Provider\" is a Python class that gets registered with the ValidMind Library and loads tests based on a test ID, for example `my_test_provider.my_test_id`. The built-in suite of tests that ValidMind offers is technically its own test provider. You can use one the built-in test provider offered by ValidMind (`validmind.tests.test_providers.LocalTestProvider`) or you can create your own. More than likely, you'll want to use the `LocalTestProvider` to add a directory of custom tests but there's flexibility to be able to load tests from any source.\n",
    "\n",
    "In this notebook, we'll take you through the process of creating a folder of custom tests from existing, inline tests. We'll then show you how to defined and register a `LocalTestProvider` that points to that folder. Finally, we'll show how the test ID works when using test providers and how you can run and use custom tests from your provider."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Contents    \n",
    "- [Prerequisites](#toc1_)    \n",
    "- [Key concepts](#toc2_)    \n",
    "- [High-level steps](#toc3_)    \n",
    "- [About ValidMind](#toc4_)    \n",
    "  - [Before you begin](#toc4_1_)    \n",
    "  - [New to ValidMind?](#toc4_2_)    \n",
    "- [Install the ValidMind Library](#toc5_)    \n",
    "- [Initialize the ValidMind Library](#toc6_)    \n",
    "  - [Get your code snippet](#toc6_1_) \n",
    "- [Set up custom tests](#toc7_)    \n",
    "- [Registering the test provider](#toc8_)    \n",
    "  - [Test providers overview](#toc8_1_)    \n",
    "  - [Local test provider](#toc8_2_)    \n",
    "  - [Running test provider tests](#toc8_3_)    \n",
    "    - [Set up the model and dataset](#toc8_3_1_)    \n",
    "    - [Run the tests](#toc8_3_2_)    \n",
    "    - [Add custom tests to model documentation](#toc8_3_3_)    \n",
    "    - [Run, save, and add other tests](#toc8_3_4_)    \n",
    "  - [Verify that `preview_template()` now loads the tests from the test providers](#toc8_4_)    \n",
    "  - [Run the documentation tests](#toc8_5_)    \n",
    "- [Conclusion](#toc9_)    \n",
    "- [Next steps](#toc10_)    \n",
    "  - [Work with your model documentation](#toc10_1_)    \n",
    "  - [Discover more learning resources](#toc10_2_)  \n",
    "- [Upgrade ValidMind](#toc11_)  \n",
    "\n",
    "<!-- vscode-jupyter-toc-config\n",
    "\tnumbering=false\n",
    "\tanchor=true\n",
    "\tflat=false\n",
    "\tminLevel=2\n",
    "\tmaxLevel=4\n",
    "\t/vscode-jupyter-toc-config -->\n",
    "<!-- THIS CELL WILL BE REPLACED ON TOC UPDATE. DO NOT WRITE YOUR TEXT IN THIS CELL -->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc1_'></a>\n",
    "\n",
    "## Prerequisites\n",
    "\n",
    "As mentioned above, its recommended that you have gone through the [Implement custom tests notebook](./implement_custom_tests.ipynb) to understand the basics of creating custom tests. We will be using the same tests defined in that notebook.\n",
    "\n",
    "If you encounter errors due to missing modules in your Python environment, install the modules with `pip install`, and then re-run the notebook. For more help, refer to [Installing Python Modules](https://docs.python.org/3/installing/index.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc2_'></a>\n",
    "\n",
    "## Key concepts\n",
    "\n",
    "- **Test Provider**: A Python class that gets registered with the ValidMind Library and loads tests based on a test ID e.g. `my_test_provider.my_test_id`.\n",
    "- **LocalTestProvider**: A built-in test provider that loads tests from a local directory.\n",
    "- **Test ID**: A unique identifier for a test that is used to load the test from a test provider. Test IDs in ValidMind have multiple parts, separated by dots (`.`), that have special meaning/functions - `namespace.path.test_name:result_id` (e.g. `validmind.data_validation.ClassImbalance:1`):\n",
    "    - `namespace`: The namespace of the test - built-in tests use the `validmind` namespace, custom test providers specify a custom namespace when they are registered\n",
    "    - `path`: The path to the test within the test provider. For internal tests, this is relative to the `validmind/tests` directory. When using a `LocalTestProvider` this is relative to the root directory you specify when creating the provider class instance. **Note: This path-based approach is a convention and not a strict requirement when implementing your own test provider.**\n",
    "    - `test_name`: The name of the test module (file) as well as the name of the function or class that is the \"test\" itself. Again, this is a convention for internal and `LocalTestProvider` tests, but test files should be named the same as the test function or class they contain.\n",
    "    - `result_id`: An optional identifier for the test. Especially useful, and required, for running tests multiple times in one section. This is separated from the previous parts by a colon `:` and can be effectively ignored when loading the test itself as its only used to uniquely identify the test result."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc3_'></a>\n",
    "\n",
    "## High-level steps\n",
    "\n",
    "1. Create a folder of custom tests from existing, inline tests.\n",
    "2. Define and register a `LocalTestProvider` that points to that folder.\n",
    "3. Use the test ID to run, view and log custom tests from your provider.\n",
    "4. Add the test results to your documentation\n",
    "5. `run_documentation_tests()` to run custom tests as part of your documentation template suite"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc4_'></a>\n",
    "\n",
    "## About ValidMind\n",
    "\n",
    "ValidMind's suite of tools enables organizations to identify, document, and manage model risks for all types of models, including AI/ML models, LLMs, and statistical models. As a model developer, you use the ValidMind Library to automate documentation and validation tests, and then use the ValidMind Platform to collaborate on model documentation. Together, these products simplify model risk management, facilitate compliance with regulations and institutional standards, and enhance collaboration between yourself and model validators.\n",
    "\n",
    "If this is your first time trying out ValidMind, we recommend going through the following resources first:\n",
    "\n",
    "- [Get started](https://docs.validmind.ai/get-started/get-started.html) — The basics, including key concepts, and how our products work\n",
    "- [ValidMind Library](https://docs.validmind.ai/developer/validmind-library.html) — The path for developers, more code samples, and our developer reference\n",
    "\n",
    "<a id='toc4_1_'></a>\n",
    "\n",
    "### Before you begin\n",
    "\n",
    "<div class=\"alert alert-block alert-info\" style=\"background-color: #B5B5B510; color: black; border: 1px solid #083E44; border-left-width: 5px; box-shadow: 2px 2px 4px rgba(0, 0, 0, 0.2);border-radius: 5px;\"><span style=\"color: #083E44;\"><b>For access to all features available in this notebook, create a free ValidMind account.</b></span>\n",
    "<br></br>\n",
    "Signing up is FREE — <a href=\"https://docs.validmind.ai/guide/configuration/register-with-validmind.html\" style=\"color: #DE257E;\"><b>Register with ValidMind</b></a></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc5_'></a>\n",
    "\n",
    "## Install the ValidMind Library\n",
    "\n",
    "To install the library:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install -q validmind"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc6_'></a>\n",
    "\n",
    "## Initialize the ValidMind Library\n",
    "\n",
    "ValidMind generates a unique _code snippet_ for each registered model to connect with your developer environment. You initialize the ValidMind Library with this code snippet, which ensures that your documentation and tests are uploaded to the correct model when you run the notebook.\n",
    "\n",
    "<a id='toc6_1_'></a>\n",
    "\n",
    "### Get your code snippet\n",
    "\n",
    "1. In a browser, [log in to ValidMind](https://docs.validmind.ai/guide/configuration/log-in-to-validmind.html).\n",
    "\n",
    "2. In the left sidebar, navigate to **Model Inventory** and click **+ Register Model**.\n",
    "\n",
    "3. Enter the model details and click **Continue**. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/register-models-in-inventory.html))\n",
    "\n",
    "   For example, to register a model for use with this notebook, select:\n",
    "\n",
    "   - Documentation template: `Binary classification`\n",
    "   - Use case: `Marketing/Sales - Attrition/Churn Management`\n",
    "\n",
    "   You can fill in other options according to your preference.\n",
    "\n",
    "4. Go to **Getting Started** and click **Copy snippet to clipboard**.\n",
    "\n",
    "Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load your model identifier credentials from an `.env` file\n",
    "\n",
    "%load_ext dotenv\n",
    "%dotenv .env\n",
    "\n",
    "# Or replace with your code snippet\n",
    "\n",
    "import validmind as vm\n",
    "\n",
    "vm.init(\n",
    "    # api_host=\"...\",\n",
    "    # api_key=\"...\",\n",
    "    # api_secret=\"...\",\n",
    "    # model=\"...\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc7_'></a>\n",
    "\n",
    "## Set up custom tests\n",
    "\n",
    "If you've gone through the [Implement custom tests notebook](./implement_custom_tests.ipynb), you should have a good understanding of how custom tests are implemented. If you haven't, we recommend going through that notebook first. In this notebook, we will take those custom tests and move them into separate modules in a folder. This is the logical progression from the previous notebook, as it allows you to take one-off tests and move them into an organized structure that makes it easier to manage, maintain and share them.\n",
    "\n",
    "First let's set the path to our custom tests folder."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tests_folder = \"my_tests\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's create this folder if it doesn't exist. If it does, let's empty it to start fresh."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "# create tests folder\n",
    "os.makedirs(tests_folder, exist_ok=True)\n",
    "\n",
    "# remove existing tests\n",
    "for f in os.listdir(tests_folder):\n",
    "    # remove files and pycache\n",
    "    if f.endswith(\".py\") or f == \"__pycache__\":\n",
    "        os.system(f\"rm -rf {tests_folder}/{f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's go ahead and redefine the custom tests we created in the previous notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "from sklearn import metrics\n",
    "from validmind import test\n",
    "\n",
    "\n",
    "@test(\"my_custom_tests.ConfusionMatrix\")\n",
    "def confusion_matrix(dataset, model):\n",
    "    \"\"\"The confusion matrix is a table that is often used to describe the performance of a classification model on a set of data for which the true values are known.\n",
    "\n",
    "    The confusion matrix is a 2x2 table that contains 4 values:\n",
    "\n",
    "    - True Positive (TP): the number of correct positive predictions\n",
    "    - True Negative (TN): the number of correct negative predictions\n",
    "    - False Positive (FP): the number of incorrect positive predictions\n",
    "    - False Negative (FN): the number of incorrect negative predictions\n",
    "\n",
    "    The confusion matrix can be used to assess the holistic performance of a classification model by showing the accuracy, precision, recall, and F1 score of the model on a single figure.\n",
    "    \"\"\"\n",
    "    confusion_matrix = metrics.confusion_matrix(dataset.y, dataset.y_pred(model))\n",
    "\n",
    "    cm_display = metrics.ConfusionMatrixDisplay(\n",
    "        confusion_matrix=confusion_matrix, display_labels=[False, True]\n",
    "    )\n",
    "    cm_display.plot()\n",
    "\n",
    "    plt.close()  # close the plot to avoid displaying it\n",
    "\n",
    "    return cm_display.figure_  # return the figure object itself"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The decorator thats used to register these as one-off custom tests adds a convenience method to the function object that allows us to simply call `<func_name>.save()` to save it to a file. This will save the function to a Python file in the current directory. You can also pass a path to a folder to save it to a different location. In our case, we can pass the variable `tests_folder` to save it to the custom tests folder we created.\n",
    "\n",
    "Normally, this will get you started by creating the file and saving the function code with the correct name. But it won't automatically add any import or other functions/variables outside of the function that are needed for the test to run. In that case, you would need to manually add those to the file.\n",
    "\n",
    "But for the sake of this demo, we will pass a list of imports into the save method to automatically add them to the file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "confusion_matrix.save(\n",
    "    tests_folder,\n",
    "    imports=[\"import matplotlib.pyplot as plt\", \"from sklearn import metrics\"],\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's go ahead and do this for the rest of the tests."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "@test(\"my_custom_tests.Hyperparameters\")\n",
    "def hyperparameters(model):\n",
    "    \"\"\"The hyperparameters of a machine learning model are the settings that control the learning process.\n",
    "    These settings are specified before the learning process begins and can have a significant impact on the\n",
    "    performance of the model.\n",
    "\n",
    "    The hyperparameters of a model can be used to tune the model to achieve the best possible performance\n",
    "    on a given dataset. By examining the hyperparameters of a model, you can gain insight into how the model\n",
    "    was trained and how it might be improved.\n",
    "    \"\"\"\n",
    "    hyperparameters = model.model.get_xgb_params()  # dictionary of hyperparameters\n",
    "\n",
    "    # turn the dictionary into a table where each row contains a hyperparameter and its value\n",
    "    return [{\"Hyperparam\": k, \"Value\": v} for k, v in hyperparameters.items() if v]\n",
    "\n",
    "\n",
    "hyperparameters.save(tests_folder)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "import random\n",
    "\n",
    "\n",
    "@test(\"my_custom_tests.ExternalAPI\")\n",
    "def external_api():\n",
    "    \"\"\"This test calls an external API to get a list of fake users. It then creates\n",
    "    a table with the relevant data so it can be displayed in the documentation.\n",
    "\n",
    "    The purpose of this test is to demonstrate how to call an external API and use the\n",
    "    data in a test. A test like this could even be setup to run in a scheduled\n",
    "    pipeline to keep your documentation in-sync with an external data source.\n",
    "    \"\"\"\n",
    "    url = \"https://jsonplaceholder.typicode.com/users\"\n",
    "    response = requests.get(url)\n",
    "    data = response.json()\n",
    "\n",
    "    # extract the time and the current BTC price in USD\n",
    "    return {\n",
    "        \"Model Owners/Stakeholders\": [\n",
    "            {\n",
    "                \"Name\": user[\"name\"],\n",
    "                \"Role\": random.choice([\"Owner\", \"Stakeholder\"]),\n",
    "                \"Email\": user[\"email\"],\n",
    "                \"Phone\": user[\"phone\"],\n",
    "                \"Slack Handle\": f\"@{user['name'].lower().replace(' ', '.')}\",\n",
    "            }\n",
    "            for user in data[:3]\n",
    "        ],\n",
    "        \"Model Developers\": [\n",
    "            {\n",
    "                \"Name\": user[\"name\"],\n",
    "                \"Role\": \"Developer\",\n",
    "                \"Email\": user[\"email\"],\n",
    "            }\n",
    "            for user in data[3:7]\n",
    "        ],\n",
    "        \"Model Validators\": [\n",
    "            {\n",
    "                \"Name\": user[\"name\"],\n",
    "                \"Role\": \"Validator\",\n",
    "                \"Email\": user[\"email\"],\n",
    "            }\n",
    "            for user in data[7:]\n",
    "        ],\n",
    "    }\n",
    "\n",
    "\n",
    "external_api.save(tests_folder, imports=[\"import requests\", \"import random\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import plotly_express as px\n",
    "\n",
    "\n",
    "@test(\"my_custom_tests.ParameterExample\")\n",
    "def parameter_example(\n",
    "    plot_title=\"Default Plot Title\", x_col=\"sepal_width\", y_col=\"sepal_length\"\n",
    "):\n",
    "    \"\"\"This test takes two parameters and creates a scatter plot based on them.\n",
    "\n",
    "    The purpose of this test is to demonstrate how to create a test that takes\n",
    "    parameters and uses them to generate a plot. This can be useful for creating\n",
    "    tests that are more flexible and can be used in a variety of scenarios.\n",
    "    \"\"\"\n",
    "    # return px.scatter(px.data.iris(), x=x_col, y=y_col, color=\"species\")\n",
    "    return px.scatter(\n",
    "        px.data.iris(), x=x_col, y=y_col, color=\"species\", title=plot_title\n",
    "    )\n",
    "\n",
    "\n",
    "parameter_example.save(tests_folder, imports=[\"import plotly_express as px\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import plotly_express as px\n",
    "\n",
    "\n",
    "@test(\"my_custom_tests.ComplexOutput\")\n",
    "def complex_output():\n",
    "    \"\"\"This test demonstrates how to return many tables and figures in a single test\"\"\"\n",
    "    # create a couple tables\n",
    "    table = [{\"A\": 1, \"B\": 2}, {\"A\": 3, \"B\": 4}]\n",
    "    table2 = [{\"C\": 5, \"D\": 6}, {\"C\": 7, \"D\": 8}]\n",
    "\n",
    "    # create a few figures showing some random data\n",
    "    fig1 = px.line(x=np.arange(10), y=np.random.rand(10), title=\"Random Line Plot\")\n",
    "    fig2 = px.bar(x=[\"A\", \"B\", \"C\"], y=np.random.rand(3), title=\"Random Bar Plot\")\n",
    "    fig3 = px.scatter(\n",
    "        x=np.random.rand(10), y=np.random.rand(10), title=\"Random Scatter Plot\"\n",
    "    )\n",
    "\n",
    "    return (\n",
    "        {\n",
    "            \"My Cool Table\": table,\n",
    "            \"Another Table\": table2,\n",
    "        },\n",
    "        fig1,\n",
    "        fig2,\n",
    "        fig3,\n",
    "    )\n",
    "\n",
    "\n",
    "complex_output.save(\n",
    "    tests_folder, imports=[\"import numpy as np\", \"import plotly_express as px\"]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import io\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "\n",
    "@test(\"my_custom_tests.Image\")\n",
    "def image():\n",
    "    \"\"\"This test demonstrates how to return an image in a test\"\"\"\n",
    "\n",
    "    # create a simple plot\n",
    "    fig, ax = plt.subplots()\n",
    "    ax.plot([1, 2, 3, 4])\n",
    "    ax.set_title(\"Simple Line Plot\")\n",
    "\n",
    "    # save the plot as a PNG image (in-memory buffer)\n",
    "    img_data = io.BytesIO()\n",
    "    fig.savefig(img_data, format=\"png\")\n",
    "    img_data.seek(0)\n",
    "\n",
    "    plt.close()  # close the plot to avoid displaying it\n",
    "\n",
    "    return img_data.read()\n",
    "\n",
    "\n",
    "image.save(tests_folder, imports=[\"import io\", \"import matplotlib.pyplot as plt\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc8_'></a>\n",
    "\n",
    "## Registering the test provider"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc8_1_'></a>\n",
    "\n",
    "### Test providers overview\n",
    "\n",
    "Now that we have a folder with our custom tests, we can create a test provider that will tell the ValidMind Library where to find these tests. ValidMind offers out-of-the-box test providers for local tests (i.e. tests in a folder) or a Github provider for tests in a Github repository. You can also create your own test provider by creating a class that has a `load_test` method that takes a test ID and returns the test function matching that ID. The protocol for test providers is below:\n",
    "\n",
    "```python\n",
    "class ExternalTestProvider(Protocol):\n",
    "    \"\"\"Protocol for user-defined test providers\"\"\"\n",
    "\n",
    "    def load_test(self, test_id: str) -> callable:\n",
    "        \"\"\"Load the test by test ID\n",
    "\n",
    "        Args:\n",
    "            test_id (str): The test ID (does not contain the namespace under which\n",
    "                the test is registered)\n",
    "\n",
    "        Returns:\n",
    "            callable: The test object\n",
    "\n",
    "        Raises:\n",
    "            FileNotFoundError: If the test is not found\n",
    "        \"\"\"\n",
    "        ...\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc8_2_'></a>\n",
    "\n",
    "### Local test provider\n",
    "\n",
    "For most use-cases, the local test provider should be sufficient. Let's go ahead and see how we can do this with our custom tests."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from validmind.tests import LocalTestProvider\n",
    "\n",
    "# initialize the test provider with the tests folder we created earlier\n",
    "my_test_provider = LocalTestProvider(tests_folder)\n",
    "\n",
    "vm.tests.register_test_provider(\n",
    "    namespace=\"my_test_provider\",\n",
    "    test_provider=my_test_provider,\n",
    ")\n",
    "# `my_test_provider.load_test()` will be called for any test ID that starts with `my_test_provider`\n",
    "# e.g. `my_test_provider.ConfusionMatrix` will look for a function named `ConfusionMatrix` in `my_tests/ConfusionMatrix.py` file"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc8_3_'></a>\n",
    "\n",
    "### Running test provider tests\n",
    "\n",
    "Now that we have our test provider set up, we can run any test that's located in our tests folder by using the `run_test()` method. This function is your entry point to running single tests in the ValidMind Library. It takes a test ID and runs the test associated with that ID. For our custom tests, the test ID will be the `namespace` specified when registering the provider, followed by the path to the test file relative to the tests folder. For example, the Confusion Matrix test we created earlier will have the test ID `my_test_provider.ConfusionMatrix`. You could organize the tests in subfolders, say `classification` and `regression`, and the test ID for the Confusion Matrix test would then be `my_test_provider.classification.ConfusionMatrix`.\n",
    "\n",
    "Let's go ahead and run some of our tests. But first, let's setup a dataset and model to run against."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc8_3_1_'></a>\n",
    "\n",
    "#### Set up the model and dataset\n",
    "\n",
    "First let's setup a an example model and dataset to run our custom metic against. Since this is a Confusion Matrix, we will use the Customer Churn dataset that ValidMind provides and train a simple XGBoost model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import xgboost as xgb\n",
    "from validmind.datasets.classification import customer_churn\n",
    "\n",
    "raw_df = customer_churn.load_data()\n",
    "train_df, validation_df, test_df = customer_churn.preprocess(raw_df)\n",
    "\n",
    "x_train = train_df.drop(customer_churn.target_column, axis=1)\n",
    "y_train = train_df[customer_churn.target_column]\n",
    "x_val = validation_df.drop(customer_churn.target_column, axis=1)\n",
    "y_val = validation_df[customer_churn.target_column]\n",
    "\n",
    "model = xgb.XGBClassifier(early_stopping_rounds=10)\n",
    "model.set_params(\n",
    "    eval_metric=[\"error\", \"logloss\", \"auc\"],\n",
    ")\n",
    "model.fit(\n",
    "    x_train,\n",
    "    y_train,\n",
    "    eval_set=[(x_val, y_val)],\n",
    "    verbose=False,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Easy enough! Now we have a model and dataset setup and trained. One last thing to do is bring the dataset and model into the ValidMind Library:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# for now, we'll just use the test dataset\n",
    "vm_raw_ds = vm.init_dataset(\n",
    "    dataset=raw_df,\n",
    "    target_column=customer_churn.target_column,\n",
    "    input_id=\"raw_dataset\",\n",
    ")\n",
    "vm_train_ds = vm.init_dataset(\n",
    "    dataset=train_df,\n",
    "    target_column=customer_churn.target_column,\n",
    "    input_id=\"train_dataset\",\n",
    ")\n",
    "vm_test_ds = vm.init_dataset(\n",
    "    dataset=test_df,\n",
    "    target_column=customer_churn.target_column,\n",
    "    input_id=\"test_dataset\",\n",
    ")\n",
    "\n",
    "vm_model = vm.init_model(model, input_id=\"model\")\n",
    "# link the model to the datasets\n",
    "vm_train_ds.assign_predictions(model=vm_model)\n",
    "vm_test_ds.assign_predictions(model=vm_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc8_3_2_'></a>\n",
    "\n",
    "#### Run the tests\n",
    "\n",
    "Now that we have our model and dataset setup, we can run our custom tests against them. Let's go ahead and run the Confusion Matrix test."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from validmind.tests import run_test\n",
    "\n",
    "result = run_test(\n",
    "    \"my_test_provider.ConfusionMatrix\",\n",
    "    inputs={\"dataset\": vm_test_ds, \"model\": vm_model},\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You should see the output of the test above. If you want to learn more about how test functions are run and get turned into test results, check out the [Implement custom tests notebook](./implement_custom_tests.ipynb).\n",
    "\n",
    "Let's go ahead and log the result to the ValidMind Platform."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "result.log()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc8_3_3_'></a>\n",
    "\n",
    "#### Add custom tests to model documentation\n",
    "\n",
    "Now that the result has been logged to the ValidMind Platform, you can add it to your model documentation. This will add the result where you specify but it also will add the test to the template so it gets run anytime you `run_documentation_tests()`. To do this, go to the documentation page of the model you connected to above and navigate to the `Model Development` -> `Model Evaluation` section. Then hover between any existing content block to reveal the `+` button as shown in the screenshot below.\n",
    "\n",
    "![screenshot showing insert button for test-driven blocks](../../images/insert-test-driven-block.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now click on the `+` button and select the `Test-Driven Block` option. This will open a dialog where you can select `My Test Provider Confusion Matrix` from the list of available tests. You can preview the result and then click `Insert Block` to add it to the documentation.\n",
    "\n",
    "![screenshot showing how to insert a test-driven block](../../images/insert-test-driven-block-test-provider.png)\n",
    "\n",
    "The test should match the result you see above."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc8_3_4_'></a>\n",
    "\n",
    "#### Run, save, and add other tests\n",
    "\n",
    "Let's take the same steps for all of our other custom tests."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "run_test(\n",
    "    \"my_test_provider.ComplexOutput\", inputs={\"dataset\": vm_test_ds, \"model\": vm_model}\n",
    ").log()\n",
    "run_test(\"my_test_provider.ExternalAPI\").log()\n",
    "run_test(\"my_test_provider.Hyperparameters\", inputs={\"model\": vm_model}).log()\n",
    "run_test(\"my_test_provider.Image\").log()\n",
    "run_test(\n",
    "    \"my_test_provider.ParameterExample\",\n",
    "    params={\n",
    "        \"plot_title\": \"Test Provider Plot\",\n",
    "        \"x_col\": \"petal_width\",\n",
    "        \"y_col\": \"petal_length\",\n",
    "    },\n",
    ").log()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that they are all saved, follow the same steps above to add them to the model documentation. Once thats done, we will run the documentation tests to see all of them run automatically."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc8_4_'></a>\n",
    "\n",
    "### Verify that `preview_template()` now loads the tests from the test providers\n",
    "\n",
    "Now that we have added the tests to the model documentation, we can run the `preview_template()` method to see the tests run automatically.\n",
    "\n",
    "First, let's reload the connection to the model to get the updated documentation template."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vm.reload()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, run `preview_template()` and verify that the tests you added are included in the proper section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vm.preview_template()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc8_5_'></a>\n",
    "\n",
    "### Run the documentation tests\n",
    "\n",
    "Now we can run the documentation tests as normal. This should include all of our custom tests which will be loaded from our test provider folder."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_config = customer_churn.get_demo_test_config()\n",
    "suite_results = vm.run_documentation_tests(config=test_config)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc9_'></a>\n",
    "\n",
    "## Conclusion\n",
    "\n",
    "In this notebook, we showed how to integrate custom tests into the ValidMind Library. We created a custom test provider that loads tests from a folder and then ran the tests against a model and dataset. We then added the tests to the model documentation and ran the documentation tests to see all of the tests run automatically. This is a powerful concept that allows you to create, organize and, most importantly, share custom tests with other model developers."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc10_'></a>\n",
    "\n",
    "## Next steps\n",
    "\n",
    "You can look at the results of this test suite right in the notebook where you ran the code, as you would expect. But there is a better way — use the ValidMind Platform to work with your model documentation.\n",
    "\n",
    "<a id='toc10_1_'></a>\n",
    "\n",
    "### Work with your model documentation\n",
    "\n",
    "1. From the **Model Inventory** in the ValidMind Platform, go to the model you registered earlier. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/working-with-model-inventory.html))\n",
    "\n",
    "2. Click and expand the **Model Development** section.\n",
    "\n",
    "What you see is the full draft of your model documentation in a more easily consumable version. From here, you can make qualitative edits to model documentation, view guidelines, collaborate with validators, and submit your model documentation for approval when it's ready. [Learn more ...](https://docs.validmind.ai/guide/model-documentation/working-with-model-documentation.html)\n",
    "\n",
    "<a id='toc10_2_'></a>\n",
    "\n",
    "### Discover more learning resources\n",
    "\n",
    "We offer many interactive notebooks to help you document models:\n",
    "\n",
    "- [Run tests & test suites](https://docs.validmind.ai/developer/model-testing/testing-overview.html)\n",
    "- [Code samples](https://docs.validmind.ai/developer/samples-jupyter-notebooks.html)\n",
    "\n",
    "Or, visit our [documentation](https://docs.validmind.ai/) to learn more about ValidMind."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc11_'></a>\n",
    "\n",
    "## Upgrade ValidMind\n",
    "\n",
    "<div class=\"alert alert-block alert-info\" style=\"background-color: #B5B5B510; color: black; border: 1px solid #083E44; border-left-width: 5px; box-shadow: 2px 2px 4px rgba(0, 0, 0, 0.2);border-radius: 5px;\">After installing ValidMind, you’ll want to periodically make sure you are on the latest version to access any new features and other enhancements.</div>\n",
    "\n",
    "Retrieve the information for the currently installed version of ValidMind:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip show validmind"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If the version returned is lower than the version indicated in our [production open-source code](https://github.com/validmind/validmind-library/blob/prod/validmind/__version__.py), restart your notebook and run:\n",
    "\n",
    "```bash\n",
    "%pip install --upgrade validmind\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You may need to restart your kernel after running the upgrade package for changes to be applied."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
