{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# ValidMind for model development 3 — Integrate custom tests\n",
    "\n",
    "Learn how to use ValidMind for your end-to-end model documentation process with our series of four introductory notebooks. In this third notebook, supplement ValidMind tests with your own and include them as additional evidence in your documentation.\n",
    "\n",
    "This notebook assumes that you already have a repository of custom made tests considered critical to include in your documentation. A custom test is any function that takes a set of inputs and parameters as arguments and returns one or more outputs:\n",
    "\n",
    "- The function can be as simple or as complex as you need it to be — it can use external libraries, make API calls, or do anything else that you can do in Python.\n",
    "- The only requirement is that the function signature and return values can be \"understood\" and handled by the ValidMind Library. As such, custom tests offer added flexibility by extending the default tests provided by ValidMind, enabling you to document any type of model or use case.\n",
    "\n",
    "**For a more in-depth introduction to custom tests,** refer to our [Implement custom tests](../../code_samples/custom_tests/implement_custom_tests.ipynb) notebook.\n",
    "\n",
    "<div class=\"alert alert-block alert-info\" style=\"background-color: #B5B5B510; color: black; border: 1px solid #083E44; border-left-width: 5px; box-shadow: 2px 2px 4px rgba(0, 0, 0, 0.2);border-radius: 5px;\"><span style=\"color: #083E44;\"><b>Learn by doing</b></span>\n",
    "<br></br>\n",
    "Our course tailor-made for developers new to ValidMind combines this series of notebooks with more a more in-depth introduction to the ValidMind Platform — <a href=\"https://docs.validmind.ai/training/developer-fundamentals/developer-fundamentals-register.html\" style=\"color: #DE257E;\"><b>Developer Fundamentals</b></a></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "::: {.content-hidden when-format=\"html\"}\n",
    "## Contents    \n",
    "- [Prerequisites](#toc1_)    \n",
    "- [Setting up](#toc2_)    \n",
    "  - [Initialize the ValidMind Library](#toc2_1_)    \n",
    "  - [Import sample dataset](#toc2_2_)    \n",
    "    - [Remove highly correlated features](#toc2_2_1_)    \n",
    "  - [Train the model](#toc2_3_)    \n",
    "    - [Initialize the ValidMind objects](#toc2_3_1_)    \n",
    "    - [Assign predictions](#toc2_3_2_)    \n",
    "- [Implementing a custom inline test](#toc3_)    \n",
    "  - [Create a confusion matrix plot](#toc3_1_)    \n",
    "  - [Add parameters to custom tests](#toc3_2_)    \n",
    "  - [Pass parameters to custom tests](#toc3_3_)    \n",
    "  - [Log the confusion matrix results](#toc3_4_)    \n",
    "- [Using external test providers](#toc4_)    \n",
    "  - [Create custom tests folder](#toc4_1_)    \n",
    "  - [Save an inline test](#toc4_2_)    \n",
    "  - [Register a local test provider](#toc4_3_)    \n",
    "    - [Initialize a local test provider](#toc4_3_1_)    \n",
    "    - [Run test provider tests](#toc4_3_2_)    \n",
    "- [Add test results to documentation](#toc5_)    \n",
    "- [In summary](#toc6_)    \n",
    "- [Next steps](#toc7_)    \n",
    "  - [Finalize testing and documentation](#toc7_1_)    \n",
    "\n",
    ":::\n",
    "<!-- jn-toc-notebook-config\n",
    "\tnumbering=false\n",
    "\tanchor=true\n",
    "\tflat=false\n",
    "\tminLevel=2\n",
    "\tmaxLevel=4\n",
    "\t/jn-toc-notebook-config -->\n",
    "<!-- THIS CELL WILL BE REPLACED ON TOC UPDATE. DO NOT WRITE YOUR TEXT IN THIS CELL -->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc1_'></a>\n",
    "\n",
    "## Prerequisites\n",
    "\n",
    "In order to integrate custom tests with your model documentation with this notebook, you'll need to first have:\n",
    "\n",
    "- [x] Registered a model within the ValidMind Platform with a predefined documentation template\n",
    "- [x] Installed the ValidMind Library in your local environment, allowing you to access all its features\n",
    "- [x] Learned how to import and initialize datasets for use with ValidMind\n",
    "- [x] Understood the basics of how to run and log tests with ValidMind\n",
    "- [x] Inserted a test-driven block for the results of your `HighPearsonCorrelation:balanced_raw_dataset` test into your model's documentation\n",
    "\n",
    "<div class=\"alert alert-block alert-info\" style=\"background-color: #B5B5B510; color: black; border: 1px solid #083E44; border-left-width: 5px; box-shadow: 2px 2px 4px rgba(0, 0, 0, 0.2);border-radius: 5px;\"><span style=\"color: #083E44;\"><b>Need help with the above steps?</b></span>\n",
    "<br></br>\n",
    "Refer to the first two notebooks in this series:\n",
    "\n",
    "- <a href=\"1-set_up_validmind.ipynb\" style=\"color: #DE257E;\"><b>1 — Set up the ValidMind Library</b></a>\n",
    "- <a href=\"2-start_development_process.ipynb\" style=\"color: #DE257E;\"><b>2 — Start the model development process</b></a>\n",
    "\n",
    "</div>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc2_'></a>\n",
    "\n",
    "## Setting up\n",
    "\n",
    "This section should be quite familiar to you — as we performed the same actions in the previous notebook, **[2 — Start the model development process](2-start_development_process.ipynb)**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc2_1_'></a>\n",
    "\n",
    "### Initialize the ValidMind Library\n",
    "\n",
    "As usual, let's first connect up the ValidMind Library to our model we previously registered in the ValidMind Platform:\n",
    "\n",
    "1. In a browser, [log in to ValidMind](https://docs.validmind.ai/guide/configuration/log-in-to-validmind.html).\n",
    "\n",
    "2. In the left sidebar, navigate to **Inventory** and select the model you registered for this \"ValidMind for model development\" series of notebooks.\n",
    "\n",
    "3. Go to **Getting Started** and click **Copy snippet to clipboard**.\n",
    "\n",
    "Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Make sure the ValidMind Library is installed\n",
    "\n",
    "%pip install -q validmind\n",
    "\n",
    "# Load your model identifier credentials from an `.env` file\n",
    "\n",
    "%load_ext dotenv\n",
    "%dotenv .env\n",
    "\n",
    "# Or replace with your code snippet\n",
    "\n",
    "import validmind as vm\n",
    "\n",
    "vm.init(\n",
    "    # api_host=\"...\",\n",
    "    # api_key=\"...\",\n",
    "    # api_secret=\"...\",\n",
    "    # model=\"...\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc2_2_'></a>\n",
    "\n",
    "### Import sample dataset\n",
    "\n",
    "Next, we'll import the same public [Bank Customer Churn Prediction](https://www.kaggle.com/datasets/shantanudhakadd/bank-customer-churn-prediction) dataset from Kaggle we used in the last notebook so that we have something to work with:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from validmind.datasets.classification import customer_churn as demo_dataset\n",
    "\n",
    "print(\n",
    "    f\"Loaded demo dataset with: \\n\\n\\t• Target column: '{demo_dataset.target_column}' \\n\\t• Class labels: {demo_dataset.class_labels}\"\n",
    ")\n",
    "\n",
    "raw_df = demo_dataset.load_data()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll apply a simple rebalancing technique to the dataset before continuing:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "raw_copy_df = raw_df.sample(frac=1)  # Create a copy of the raw dataset\n",
    "\n",
    "# Create a balanced dataset with the same number of exited and not exited customers\n",
    "exited_df = raw_copy_df.loc[raw_copy_df[\"Exited\"] == 1]\n",
    "not_exited_df = raw_copy_df.loc[raw_copy_df[\"Exited\"] == 0].sample(n=exited_df.shape[0])\n",
    "\n",
    "balanced_raw_df = pd.concat([exited_df, not_exited_df])\n",
    "balanced_raw_df = balanced_raw_df.sample(frac=1, random_state=42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc2_2_1_'></a>\n",
    "\n",
    "#### Remove highly correlated features\n",
    "\n",
    "Let's also quickly remove highly correlated features from the dataset using the output from a ValidMind test.\n",
    "\n",
    "As you learned previously, before we can run tests you'll need to initialize a ValidMind dataset object:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Register new data and now 'balanced_raw_dataset' is the new dataset object of interest\n",
    "vm_balanced_raw_dataset = vm.init_dataset(\n",
    "    dataset=balanced_raw_df,\n",
    "    input_id=\"balanced_raw_dataset\",\n",
    "    target_column=\"Exited\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With our balanced dataset initialized, we can then run our test and utilize the output to help us identify the features we want to remove:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run HighPearsonCorrelation test with our balanced dataset as input and return a result object\n",
    "corr_result = vm.tests.run_test(\n",
    "    test_id=\"validmind.data_validation.HighPearsonCorrelation\",\n",
    "    params={\"max_threshold\": 0.3},\n",
    "    inputs={\"dataset\": vm_balanced_raw_dataset},\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# From result object, extract table from `corr_result.tables`\n",
    "features_df = corr_result.tables[0].data\n",
    "features_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Extract list of features that failed the test\n",
    "high_correlation_features = features_df[features_df[\"Pass/Fail\"] == \"Fail\"][\"Columns\"].tolist()\n",
    "high_correlation_features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Extract feature names from the list of strings\n",
    "high_correlation_features = [feature.split(\",\")[0].strip(\"()\") for feature in high_correlation_features]\n",
    "high_correlation_features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can then re-initialize the dataset with a different `input_id` and the highly correlated features removed and re-run the test for confirmation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Remove the highly correlated features from the dataset\n",
    "balanced_raw_no_age_df = balanced_raw_df.drop(columns=high_correlation_features)\n",
    "\n",
    "# Re-initialize the dataset object\n",
    "vm_raw_dataset_preprocessed = vm.init_dataset(\n",
    "    dataset=balanced_raw_no_age_df,\n",
    "    input_id=\"raw_dataset_preprocessed\",\n",
    "    target_column=\"Exited\",\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Re-run the test with the reduced feature set\n",
    "corr_result = vm.tests.run_test(\n",
    "    test_id=\"validmind.data_validation.HighPearsonCorrelation\",\n",
    "    params={\"max_threshold\": 0.3},\n",
    "    inputs={\"dataset\": vm_raw_dataset_preprocessed},\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc2_3_'></a>\n",
    "\n",
    "### Train the model\n",
    "\n",
    "We'll then use ValidMind tests to train a simple logistic regression model on our prepared dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# First encode the categorical features in our dataset with the highly correlated features removed\n",
    "balanced_raw_no_age_df = pd.get_dummies(\n",
    "    balanced_raw_no_age_df, columns=[\"Geography\", \"Gender\"], drop_first=True\n",
    ")\n",
    "balanced_raw_no_age_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Split the processed dataset into train and test\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "train_df, test_df = train_test_split(balanced_raw_no_age_df, test_size=0.20)\n",
    "\n",
    "X_train = train_df.drop(\"Exited\", axis=1)\n",
    "y_train = train_df[\"Exited\"]\n",
    "X_test = test_df.drop(\"Exited\", axis=1)\n",
    "y_test = test_df[\"Exited\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "# Logistic Regression grid params\n",
    "log_reg_params = {\n",
    "    \"penalty\": [\"l1\", \"l2\"],\n",
    "    \"C\": [0.001, 0.01, 0.1, 1, 10, 100, 1000],\n",
    "    \"solver\": [\"liblinear\"],\n",
    "}\n",
    "\n",
    "# Grid search for Logistic Regression\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "grid_log_reg = GridSearchCV(LogisticRegression(), log_reg_params)\n",
    "grid_log_reg.fit(X_train, y_train)\n",
    "\n",
    "# Logistic Regression best estimator\n",
    "log_reg = grid_log_reg.best_estimator_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc2_3_1_'></a>\n",
    "\n",
    "#### Initialize the ValidMind objects\n",
    "\n",
    "Let's initialize the ValidMind `Dataset` and `Model` objects in preparation for assigning model predictions to each dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize the datasets into their own dataset objects\n",
    "vm_train_ds = vm.init_dataset(\n",
    "    input_id=\"train_dataset_final\",\n",
    "    dataset=train_df,\n",
    "    target_column=\"Exited\",\n",
    ")\n",
    "\n",
    "vm_test_ds = vm.init_dataset(\n",
    "    input_id=\"test_dataset_final\",\n",
    "    dataset=test_df,\n",
    "    target_column=\"Exited\",\n",
    ")\n",
    "\n",
    "# Initialize a model object\n",
    "vm_model = vm.init_model(log_reg, input_id=\"log_reg_model_v1\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc2_3_2_'></a>\n",
    "\n",
    "#### Assign predictions\n",
    "\n",
    "Once the model is registered, we'll assign predictions to the training and test datasets:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vm_train_ds.assign_predictions(model=vm_model)\n",
    "vm_test_ds.assign_predictions(model=vm_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc3_'></a>\n",
    "\n",
    "## Implementing a custom inline test\n",
    "\n",
    "With the set up out of the way, let's implement a custom *inline test* that calculates the confusion matrix for a binary classification model.\n",
    "\n",
    "- An inline test refers to a test written and executed within the same environment as the code being tested — in this case, right in this Jupyter Notebook —  without requiring a separate test file or framework.\n",
    "- You'll note that the custom test function is just a regular Python function that can include and require any Python library as you see fit."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc3_1_'></a>\n",
    "\n",
    "### Create a confusion matrix plot\n",
    "\n",
    "Let's first create a confusion matrix plot using the `confusion_matrix` function from the `sklearn.metrics` module:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "metadata": {}
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "from sklearn import metrics\n",
    "\n",
    "# Get the predicted classes\n",
    "y_pred = log_reg.predict(vm_test_ds.x)\n",
    "\n",
    "confusion_matrix = metrics.confusion_matrix(y_test, y_pred)\n",
    "\n",
    "cm_display = metrics.ConfusionMatrixDisplay(\n",
    "    confusion_matrix=confusion_matrix, display_labels=[False, True]\n",
    ")\n",
    "cm_display.plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, create a [`@vm.test` wrapper](https://docs.validmind.ai/validmind/validmind.html#test) that will allow you to create a reusable test. **Note the following changes in the code below:**\n",
    "\n",
    "- The function `confusion_matrix` takes two arguments `dataset` and `model`. This is a `VMDataset` and `VMModel` object respectively.\n",
    "  - `VMDataset` objects allow you to access the dataset's true (target) values by accessing the `.y` attribute.\n",
    "  - `VMDataset` objects allow you to access the predictions for a given model by accessing the `.y_pred()` method.\n",
    "- The function docstring provides a description of what the test does. This will be displayed along with the result in this notebook as well as in the ValidMind Platform.\n",
    "- The function body calculates the confusion matrix using the `sklearn.metrics.confusion_matrix` function as we just did above.\n",
    "- The function then returns the `ConfusionMatrixDisplay.figure_` object — this is important as the ValidMind Library expects the output of the custom test to be a plot or a table.\n",
    "- The `@vm.test` decorator is doing the work of creating a wrapper around the function that will allow it to be run by the ValidMind Library. It also registers the test so it can be found by the ID `my_custom_tests.ConfusionMatrix`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "metadata": {}
   },
   "outputs": [],
   "source": [
    "@vm.test(\"my_custom_tests.ConfusionMatrix\")\n",
    "def confusion_matrix(dataset, model):\n",
    "    \"\"\"The confusion matrix is a table that is often used to describe the performance of a classification model on a set of data for which the true values are known.\n",
    "\n",
    "    The confusion matrix is a 2x2 table that contains 4 values:\n",
    "\n",
    "    - True Positive (TP): the number of correct positive predictions\n",
    "    - True Negative (TN): the number of correct negative predictions\n",
    "    - False Positive (FP): the number of incorrect positive predictions\n",
    "    - False Negative (FN): the number of incorrect negative predictions\n",
    "\n",
    "    The confusion matrix can be used to assess the holistic performance of a classification model by showing the accuracy, precision, recall, and F1 score of the model on a single figure.\n",
    "    \"\"\"\n",
    "    y_true = dataset.y\n",
    "    y_pred = dataset.y_pred(model=model)\n",
    "\n",
    "    confusion_matrix = metrics.confusion_matrix(y_true, y_pred)\n",
    "\n",
    "    cm_display = metrics.ConfusionMatrixDisplay(\n",
    "        confusion_matrix=confusion_matrix, display_labels=[False, True]\n",
    "    )\n",
    "    cm_display.plot()\n",
    "\n",
    "    plt.close()  # close the plot to avoid displaying it\n",
    "\n",
    "    return cm_display.figure_  # return the figure object itself"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can now run the newly created custom test on both the training and test datasets using the [`run_test()` function](https://docs.validmind.ai/validmind/validmind/tests.html#run_test):\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "metadata": {}
   },
   "outputs": [],
   "source": [
    "# Training dataset\n",
    "result = vm.tests.run_test(\n",
    "    \"my_custom_tests.ConfusionMatrix:training_dataset\",\n",
    "    inputs={\"model\": vm_model, \"dataset\": vm_train_ds},\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Test dataset\n",
    "result = vm.tests.run_test(\n",
    "    \"my_custom_tests.ConfusionMatrix:test_dataset\",\n",
    "    inputs={\"model\": vm_model, \"dataset\": vm_test_ds},\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc3_2_'></a>\n",
    "\n",
    "### Add parameters to custom tests\n",
    "\n",
    "Custom tests can take parameters just like any other function. To demonstrate, let's modify the `confusion_matrix` function to take an additional parameter `normalize` that will allow you to normalize the confusion matrix:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "@vm.test(\"my_custom_tests.ConfusionMatrix\")\n",
    "def confusion_matrix(dataset, model, normalize=False):\n",
    "    \"\"\"The confusion matrix is a table that is often used to describe the performance of a classification model on a set of data for which the true values are known.\n",
    "\n",
    "    The confusion matrix is a 2x2 table that contains 4 values:\n",
    "\n",
    "    - True Positive (TP): the number of correct positive predictions\n",
    "    - True Negative (TN): the number of correct negative predictions\n",
    "    - False Positive (FP): the number of incorrect positive predictions\n",
    "    - False Negative (FN): the number of incorrect negative predictions\n",
    "\n",
    "    The confusion matrix can be used to assess the holistic performance of a classification model by showing the accuracy, precision, recall, and F1 score of the model on a single figure.\n",
    "    \"\"\"\n",
    "    y_true = dataset.y\n",
    "    y_pred = dataset.y_pred(model=model)\n",
    "\n",
    "    if normalize:\n",
    "        confusion_matrix = metrics.confusion_matrix(y_true, y_pred, normalize=\"all\")\n",
    "    else:\n",
    "        confusion_matrix = metrics.confusion_matrix(y_true, y_pred)\n",
    "\n",
    "    cm_display = metrics.ConfusionMatrixDisplay(\n",
    "        confusion_matrix=confusion_matrix, display_labels=[False, True]\n",
    "    )\n",
    "    cm_display.plot()\n",
    "\n",
    "    plt.close()  # close the plot to avoid displaying it\n",
    "\n",
    "    return cm_display.figure_  # return the figure object itself"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc3_3_'></a>\n",
    "\n",
    "### Pass parameters to custom tests\n",
    "\n",
    "You can pass parameters to custom tests by providing a dictionary of parameters to the `run_test()` function.\n",
    "\n",
    "- The parameters will override any default parameters set in the custom test definition. Note that `dataset` and `model` are still passed as `inputs`.\n",
    "- Since these are `VMDataset` or `VMModel` inputs, they have a special meaning.\n",
    "- When declaring a `dataset`, `model`, `datasets` or `models` argument in a custom test function, the ValidMind Library will expect these get passed as `inputs` to `run_test()` or `run_documentation_tests()`.\n",
    "\n",
    "Re-running the confusion matrix with `normalize=True` and our testing dataset looks like this:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Test dataset with normalize=True\n",
    "result = vm.tests.run_test(\n",
    "    \"my_custom_tests.ConfusionMatrix:test_dataset_normalized\",\n",
    "    inputs={\"model\": vm_model, \"dataset\": vm_test_ds},\n",
    "    params={\"normalize\": True}\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc3_4_'></a>\n",
    "\n",
    "### Log the confusion matrix results\n",
    "\n",
    "As we learned in **[2 — Start the model development process](2-start_development_process.ipynb)** under **Documenting results** > **Run and log an individual tests**, you can log any result to the ValidMind Platform with the [`.log()` method](https://docs.validmind.ai/validmind/validmind/vm_models.html#TestResult.log) of the result object, allowing you to then add the result to the documentation.\n",
    "\n",
    "You can now do the same for the confusion matrix results:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "metadata": {}
   },
   "outputs": [],
   "source": [
    "result.log()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\" style=\"background-color: #B5B5B510; color: black; border: 1px solid #083E44; border-left-width: 5px; box-shadow: 2px 2px 4px rgba(0, 0, 0, 0.2);border-radius: 5px;\"><span style=\"color: #083E44;\"><b>Note the output returned indicating that a test-driven block doesn't currently exist in your model's documentation for this particular test ID. </b></span>\n",
    "<br></br>\n",
    "That's expected, as when we run individual tests the results logged need to be manually added to your documentation within the ValidMind Platform.</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc4_'></a>\n",
    "\n",
    "## Using external test providers\n",
    "\n",
    "Creating inline custom tests with a function is a great way to customize your model documentation. However, sometimes you may want to reuse the same set of tests across multiple models and share them with others in your organization. In this case, you can create an external custom *test provider* that will allow you to load custom tests from a local folder or a Git repository.\n",
    "\n",
    "In this section you will learn how to declare a local filesystem test provider that allows loading tests from a local folder following these high level steps:\n",
    "\n",
    "1. Create a folder of custom tests from existing inline tests (tests that exist in your active Jupyter Notebook)\n",
    "2. Save an inline test to a file\n",
    "3. Define and register a [`LocalTestProvider`](https://docs.validmind.ai/validmind/validmind/tests.html#LocalTestProvider) that points to that folder\n",
    "4. Run test provider tests\n",
    "5. Add the test results to your documentation\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc4_1_'></a>\n",
    "\n",
    "### Create custom tests folder\n",
    "\n",
    "Let's start by creating a new folder that will contain reusable custom tests from your existing inline tests.\n",
    "\n",
    "The following code snippet will create a new `my_tests` directory in the current working directory if it doesn't exist:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "metadata": {}
   },
   "outputs": [],
   "source": [
    "tests_folder = \"my_tests\"\n",
    "\n",
    "import os\n",
    "\n",
    "# create tests folder\n",
    "os.makedirs(tests_folder, exist_ok=True)\n",
    "\n",
    "# remove existing tests\n",
    "for f in os.listdir(tests_folder):\n",
    "    # remove files and pycache\n",
    "    if f.endswith(\".py\") or f == \"__pycache__\":\n",
    "        os.system(f\"rm -rf {tests_folder}/{f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After running the command above, confirm that a new `my_tests` directory was created successfully. For example:\n",
    "\n",
    "```\n",
    "~/notebooks/tutorials/model_development/my_tests/\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc4_2_'></a>\n",
    "\n",
    "### Save an inline test\n",
    "\n",
    "The `@vm.test` decorator we used in **Implementing a custom inline test** above to register one-off custom tests also includes a convenience method on the function object that allows you to simply call `<func_name>.save()` to save the test to a Python file at a specified path.\n",
    "\n",
    "While `save()` will get you started by creating the file and saving the function code with the correct name, it won't automatically include any imports, or other functions or variables, outside of the functions that are needed for the test to run. To solve this, pass in an optional `imports` argument ensuring necessary imports are added to the file.\n",
    "\n",
    "The `confusion_matrix` test requires the following additional imports:\n",
    "\n",
    "```python\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn import metrics\n",
    "```\n",
    "\n",
    "Let's pass these imports to the `save()` method to ensure they are included in the file with the following command:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "metadata": {}
   },
   "outputs": [],
   "source": [
    "confusion_matrix.save(\n",
    "    # Save it to the custom tests folder we created\n",
    "    tests_folder,\n",
    "    imports=[\"import matplotlib.pyplot as plt\", \"from sklearn import metrics\"],\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- [x] Confirm that the `save()` method saved the `confusion_matrix` function to a file named `ConfusionMatrix.py` in the `my_tests` folder.\n",
    "- [x] Note that the new file provides some context on the origin of the test, which is useful for traceability:\n",
    "\n",
    "    ```\n",
    "    # Saved from __main__.confusion_matrix\n",
    "    # Original Test ID: my_custom_tests.ConfusionMatrix\n",
    "    # New Test ID: <test_provider_namespace>.ConfusionMatrix\n",
    "    ```\n",
    "\n",
    "- [x]  Additionally, the new test function has been stripped off its decorator, as it now resides in a file that will be loaded by the test provider:\n",
    "\n",
    "    ```python\n",
    "    def ConfusionMatrix(dataset, model, normalize=False):\n",
    "    ```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc4_3_'></a>\n",
    "\n",
    "### Register a local test provider\n",
    "\n",
    "Now that your `my_tests` folder has a sample custom test, let's initialize a test provider that will tell the ValidMind Library where to find your custom tests:\n",
    "\n",
    "- ValidMind offers out-of-the-box test providers for local tests (tests in a folder) or a Github provider for tests in a Github repository.\n",
    "- You can also create your own test provider by creating a class that has a [`load_test` method](https://docs.validmind.ai/validmind/validmind/tests.html#load_test) that takes a test ID and returns the test function matching that ID.\n",
    "\n",
    "<div class=\"alert alert-block alert-info\" style=\"background-color: #B5B5B510; color: black; border: 1px solid #083E44; border-left-width: 5px; box-shadow: 2px 2px 4px rgba(0, 0, 0, 0.2);border-radius: 5px;\"><span style=\"color: #083E44;\"><b>Want to learn more about test providers?</b></span>\n",
    "<br></br>\n",
    "An extended introduction to test providers can be found in: <a href=\"https://docs.validmind.ai/notebooks/code_samples/custom_tests/integrate_external_test_providers.html\" style=\"color: #DE257E;\"><b>Integrate external test providers</b></a></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc4_3_1_'></a>\n",
    "\n",
    "#### Initialize a local test provider\n",
    "\n",
    "For most use cases, using a `LocalTestProvider` that allows you to load custom tests from a designated directory should be sufficient.\n",
    "\n",
    "**The most important attribute for a test provider is its `namespace`.** This is a string that will be used to prefix test IDs in model documentation. This allows you to have multiple test providers with tests that can even share the same ID, but are distinguished by their namespace.\n",
    "\n",
    "Let's go ahead and load the custom tests from our `my_tests` directory:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from validmind.tests import LocalTestProvider\n",
    "\n",
    "# initialize the test provider with the tests folder we created earlier\n",
    "my_test_provider = LocalTestProvider(tests_folder)\n",
    "\n",
    "vm.tests.register_test_provider(\n",
    "    namespace=\"my_test_provider\",\n",
    "    test_provider=my_test_provider,\n",
    ")\n",
    "# `my_test_provider.load_test()` will be called for any test ID that starts with `my_test_provider`\n",
    "# e.g. `my_test_provider.ConfusionMatrix` will look for a function named `ConfusionMatrix` in `my_tests/ConfusionMatrix.py` file"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc4_3_2_'></a>\n",
    "\n",
    "#### Run test provider tests\n",
    "\n",
    "Now that we've set up the test provider, we can run any test that's located in the tests folder by using the `run_test()` method as with any other test:\n",
    "\n",
    "- For tests that reside in a test provider directory, the test ID will be the `namespace` specified when registering the provider, followed by the path to the test file relative to the tests folder.\n",
    "- For example, the Confusion Matrix test we created earlier will have the test ID `my_test_provider.ConfusionMatrix`. You could organize the tests in subfolders, say `classification` and `regression`, and the test ID for the Confusion Matrix test would then be `my_test_provider.classification.ConfusionMatrix`.\n",
    "\n",
    "Let's go ahead and re-run the confusion matrix test with our testing dataset by using the test ID `my_test_provider.ConfusionMatrix`. This should load the test from the test provider and run it as before.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "metadata": {}
   },
   "outputs": [],
   "source": [
    "result = vm.tests.run_test(\n",
    "    \"my_test_provider.ConfusionMatrix\",\n",
    "    inputs={\"model\": vm_model, \"dataset\": vm_test_ds},\n",
    "    params={\"normalize\": True},\n",
    ")\n",
    "\n",
    "result.log()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\" style=\"background-color: #B5B5B510; color: black; border: 1px solid #083E44; border-left-width: 5px; box-shadow: 2px 2px 4px rgba(0, 0, 0, 0.2);border-radius: 5px;\"><span style=\"color: #083E44;\"><b>Again, note the output returned indicating that a test-driven block doesn't currently exist in your model's documentation for this particular test ID. </b></span>\n",
    "<br></br>\n",
    "That's expected, as when we run individual tests the results logged need to be manually added to your documentation within the ValidMind Platform.</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc5_'></a>\n",
    "\n",
    "## Add test results to documentation\n",
    "\n",
    "With our custom tests run and results logged to the ValidMind Platform, let's head to the model we connected to at the beginning of this notebook and insert our test results into the documentation ([Need more help?](https://docs.validmind.ai/developer/model-documentation/work-with-test-results.html)):\n",
    "\n",
    "1. From the **Inventory** in the ValidMind Platform, go to the model you connected to earlier.\n",
    "\n",
    "2. In the left sidebar that appears for your model, click **Documentation**.\n",
    "\n",
    "3. Locate the Data Preparation section and click on **3.2. Model Evaluation** to expand that section.\n",
    "\n",
    "4. Hover under the Pearson Correlation Matrix content block until a horizontal dashed line with a **+** button appears, indicating that you can insert a new block.\n",
    "\n",
    "    <img src= \"add-content-block.gif\" alt=\"Screenshot showing insert block button in model documentation\" style=\"border: 2px solid #083E44; border-radius: 8px; border-right-width: 2px; border-bottom-width: 3px;\">\n",
    "    <br><br>\n",
    "\n",
    "5. Click **+** and then select **Test-Driven Block** under FROM LIBRARY:\n",
    "\n",
    "    - Click on **Custom** under TEST-DRIVEN in the left sidebar.\n",
    "    - Select the two custom `ConfusionMatrix` tests you logged above:\n",
    "\n",
    "    <img src= \"selecting-confusion-matrix-test.png\" alt=\"Screenshot showing the ConfusionMatrix tests selected\" style=\"border: 2px solid #083E44; border-radius: 8px; border-right-width: 2px; border-bottom-width: 3px;\">\n",
    "    <br><br>\n",
    "\n",
    "6. Finally, click **Insert 2 Test Results to Document** to add the test results to the documentation.\n",
    "\n",
    "    Confirm that the two individual results for the confusion matrix tests have been correctly inserted into section **3.2. Model Evaluation** of the documentation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc6_'></a>\n",
    "\n",
    "## In summary\n",
    "\n",
    "In this third notebook, you learned how to:\n",
    "\n",
    "- [x] Implement a custom inline test\n",
    "- [x] Run and log your custom inline tests\n",
    "- [x] Use external custom test providers\n",
    "- [x] Run and log tests from your custom test providers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc7_1_'></a>\n",
    "\n",
    "## Next steps"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc7_1_'></a>\n",
    "\n",
    "### Finalize testing and documentation\n",
    "\n",
    "Now that you're proficient at using the ValidMind Library to run and log tests, let's put the last pieces in place to prepare our fully documented sample model for review: **[4 — Finalize testing and documentation](4-finalize_testing_documentation.ipynb)**"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "ValidMind Library",
   "language": "python",
   "name": "validmind"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
