{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# ValidMind for model validation 4 — Finalize testing and reporting\n",
    "\n",
    "Learn how to use ValidMind for your end-to-end model validation process with our series of four introductory notebooks. In this last notebook, finalize the compliance assessment process and have a complete validation report ready for review.\n",
    "\n",
    "This notebook will walk you through how to supplement ValidMind tests with your own custom tests and include them as additional evidence in your validation report. A custom test is any function that takes a set of inputs and parameters as arguments and returns one or more outputs:\n",
    "\n",
    "- The function can be as simple or as complex as you need it to be — it can use external libraries, make API calls, or do anything else that you can do in Python.\n",
    "- The only requirement is that the function signature and return values can be \"understood\" and handled by the ValidMind Library. As such, custom tests offer added flexibility by extending the default tests provided by ValidMind, enabling you to document any type of model or use case.\n",
    "\n",
    "**For a more in-depth introduction to custom tests,** refer to our [Implement custom tests](../../code_samples/custom_tests/implement_custom_tests.ipynb) notebook.\n",
    "\n",
    "<div class=\"alert alert-block alert-info\" style=\"background-color: #B5B5B510; color: black; border: 1px solid #083E44; border-left-width: 5px; box-shadow: 2px 2px 4px rgba(0, 0, 0, 0.2);border-radius: 5px;\"><span style=\"color: #083E44;\"><b>Learn by doing</b></span>\n",
    "<br></br>\n",
    "Our course tailor-made for validators new to ValidMind combines this series of notebooks with more a more in-depth introduction to the ValidMind Platform — <a href=\"https://docs.validmind.ai/training/validator-fundamentals/validator-fundamentals-register.html\" style=\"color: #DE257E;\"><b>Validator Fundamentals</b></a></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "::: {.content-hidden when-format=\"html\"}\n",
    "## Contents    \n",
    "- [Prerequisites](#toc1_)    \n",
    "- [Setting up](#toc2_)    \n",
    "  - [Initialize the ValidMind Library](#toc2_1_)    \n",
    "  - [Import the sample dataset](#toc2_2_)    \n",
    "  - [Split the preprocessed dataset](#toc2_3_)    \n",
    "  - [Import the champion model](#toc2_4_)    \n",
    "  - [Train potential challenger model](#toc2_5_)    \n",
    "  - [Initialize the model objects](#toc2_6_)    \n",
    "- [Implementing custom tests](#toc3_)    \n",
    "  - [Implement a custom inline test](#toc3_1_)    \n",
    "    - [Create a confusion matrix plot](#toc3_1_1_)    \n",
    "    - [Add parameters to custom tests](#toc3_1_2_)    \n",
    "    - [Pass parameters to custom tests](#toc3_1_3_)    \n",
    "  - [Use external test providers](#toc3_2_)    \n",
    "    - [Create custom tests folder](#toc3_2_1_)    \n",
    "    - [Save an inline test](#toc3_2_2_)    \n",
    "    - [Register a local test provider](#toc3_2_3_)    \n",
    "- [Verify test runs](#toc4_)    \n",
    "- [In summary](#toc5_)    \n",
    "- [Next steps](#toc6_)    \n",
    "  - [Work with your validation report](#toc6_1_)    \n",
    "  - [Learn more](#toc6_2_)    \n",
    "    - [More how-to guides and code samples](#toc6_2_1_)    \n",
    "    - [Discover more learning resources](#toc6_2_2_)    \n",
    "\n",
    ":::\n",
    "<!-- jn-toc-notebook-config\n",
    "\tnumbering=false\n",
    "\tanchor=true\n",
    "\tflat=false\n",
    "\tminLevel=2\n",
    "\tmaxLevel=4\n",
    "\t/jn-toc-notebook-config -->\n",
    "<!-- THIS CELL WILL BE REPLACED ON TOC UPDATE. DO NOT WRITE YOUR TEXT IN THIS CELL -->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc1_'></a>\n",
    "\n",
    "## Prerequisites\n",
    "\n",
    "In order to finalize validation and reporting, you'll need to first have:\n",
    "\n",
    "- [x] Registered a model within the ValidMind Platform and granted yourself access to the model as a validator\n",
    "- [x] Installed the ValidMind Library in your local environment, allowing you to access all its features\n",
    "- [x] Learned how to import and initialize datasets and models for use with ValidMind\n",
    "- [x] Understood the basics of how to identify and run validation tests\n",
    "- [x] Run validation tests for your champion and challenger models, and logged the results of those tests to the ValidMind Platform\n",
    "- [x] Inserted your logged test results into your validation report\n",
    "- [x] Added some preliminary findings to your validation report\n",
    "\n",
    "<div class=\"alert alert-block alert-info\" style=\"background-color: #B5B5B510; color: black; border: 1px solid #083E44; border-left-width: 5px; box-shadow: 2px 2px 4px rgba(0, 0, 0, 0.2);border-radius: 5px;\"><span style=\"color: #083E44;\"><b>Need help with the above steps?</b></span>\n",
    "<br></br>\n",
    "Refer to the first three notebooks in this series:\n",
    "\n",
    "- <a href=\"1-set_up_validmind_for_validation.ipynb\" style=\"color: #DE257E;\"><b>1 — Set up the ValidMind Library for validation</b></a>\n",
    "- <a href=\"2-start_validation_process.ipynb\" style=\"color: #DE257E;\"><b>2 — Start the model validation process</b></a>\n",
    "- <a href=\"3-developing_challenger_model.ipynb\" style=\"color: #DE257E;\"><b>2 — Developing a potential challenger model</b></a>\n",
    "\n",
    "</div>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc2_'></a>\n",
    "\n",
    "## Setting up\n",
    "\n",
    "This section should be very familiar to you now — as we performed the same actions in the previous two notebooks in this series."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc2_1_'></a>\n",
    "\n",
    "### Initialize the ValidMind Library\n",
    "\n",
    "As usual, let's first connect up the ValidMind Library to our model we previously registered in the ValidMind Platform:\n",
    "\n",
    "1. In a browser, [log in to ValidMind](https://docs.validmind.ai/guide/configuration/log-in-to-validmind.html).\n",
    "\n",
    "2. In the left sidebar, navigate to **Inventory** and select the model you registered for this \"ValidMind for model validation\" series of notebooks.\n",
    "\n",
    "3. Go to **Getting Started** and click **Copy snippet to clipboard**.\n",
    "\n",
    "Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Make sure the ValidMind Library is installed\n",
    "\n",
    "%pip install -q validmind\n",
    "\n",
    "# Load your model identifier credentials from an `.env` file\n",
    "\n",
    "%load_ext dotenv\n",
    "%dotenv .env\n",
    "\n",
    "# Or replace with your code snippet\n",
    "\n",
    "import validmind as vm\n",
    "\n",
    "vm.init(\n",
    "    # api_host=\"...\",\n",
    "    # api_key=\"...\",\n",
    "    # api_secret=\"...\",\n",
    "    # model=\"...\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc2_2_'></a>\n",
    "\n",
    "### Import the sample dataset\n",
    "\n",
    "Next, we'll load in the same sample [Bank Customer Churn Prediction](https://www.kaggle.com/datasets/shantanudhakadd/bank-customer-churn-prediction) dataset used to develop the champion model that we will independently preprocess:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load the sample dataset\n",
    "from validmind.datasets.classification import customer_churn as demo_dataset\n",
    "\n",
    "print(\n",
    "    f\"Loaded demo dataset with: \\n\\n\\t• Target column: '{demo_dataset.target_column}' \\n\\t• Class labels: {demo_dataset.class_labels}\"\n",
    ")\n",
    "\n",
    "raw_df = demo_dataset.load_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize the raw dataset for use in ValidMind tests\n",
    "vm_raw_dataset = vm.init_dataset(\n",
    "    dataset=raw_df,\n",
    "    input_id=\"raw_dataset\",\n",
    "    target_column=\"Exited\",\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "raw_copy_df = raw_df.sample(frac=1)  # Create a copy of the raw dataset\n",
    "\n",
    "# Create a balanced dataset with the same number of exited and not exited customers\n",
    "exited_df = raw_copy_df.loc[raw_copy_df[\"Exited\"] == 1]\n",
    "not_exited_df = raw_copy_df.loc[raw_copy_df[\"Exited\"] == 0].sample(n=exited_df.shape[0])\n",
    "\n",
    "balanced_raw_df = pd.concat([exited_df, not_exited_df])\n",
    "balanced_raw_df = balanced_raw_df.sample(frac=1, random_state=42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let’s also quickly remove highly correlated features from the dataset using the output from a ValidMind test:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Register new data and now 'balanced_raw_dataset' is the new dataset object of interest\n",
    "vm_balanced_raw_dataset = vm.init_dataset(\n",
    "    dataset=balanced_raw_df,\n",
    "    input_id=\"balanced_raw_dataset\",\n",
    "    target_column=\"Exited\",\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run HighPearsonCorrelation test with our balanced dataset as input and return a result object\n",
    "corr_result = vm.tests.run_test(\n",
    "    test_id=\"validmind.data_validation.HighPearsonCorrelation\",\n",
    "    params={\"max_threshold\": 0.3},\n",
    "    inputs={\"dataset\": vm_balanced_raw_dataset},\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# From result object, extract table from `corr_result.tables`\n",
    "features_df = corr_result.tables[0].data\n",
    "features_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Extract list of features that failed the test\n",
    "high_correlation_features = features_df[features_df[\"Pass/Fail\"] == \"Fail\"][\"Columns\"].tolist()\n",
    "high_correlation_features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Extract feature names from the list of strings\n",
    "high_correlation_features = [feature.split(\",\")[0].strip(\"()\") for feature in high_correlation_features]\n",
    "high_correlation_features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Remove the highly correlated features from the dataset\n",
    "balanced_raw_no_age_df = balanced_raw_df.drop(columns=high_correlation_features)\n",
    "\n",
    "# Re-initialize the dataset object\n",
    "vm_raw_dataset_preprocessed = vm.init_dataset(\n",
    "    dataset=balanced_raw_no_age_df,\n",
    "    input_id=\"raw_dataset_preprocessed\",\n",
    "    target_column=\"Exited\",\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Re-run the test with the reduced feature set\n",
    "corr_result = vm.tests.run_test(\n",
    "    test_id=\"validmind.data_validation.HighPearsonCorrelation\",\n",
    "    params={\"max_threshold\": 0.3},\n",
    "    inputs={\"dataset\": vm_raw_dataset_preprocessed},\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc2_3_'></a>\n",
    "\n",
    "### Split the preprocessed dataset\n",
    "\n",
    "With our raw dataset rebalanced with highly correlated features removed, let's now **spilt our dataset into train and test** in preparation for model evaluation testing:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Encode categorical features in the dataset\n",
    "balanced_raw_no_age_df = pd.get_dummies(\n",
    "    balanced_raw_no_age_df, columns=[\"Geography\", \"Gender\"], drop_first=True\n",
    ")\n",
    "balanced_raw_no_age_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "# Split the dataset into train and test\n",
    "train_df, test_df = train_test_split(balanced_raw_no_age_df, test_size=0.20)\n",
    "\n",
    "X_train = train_df.drop(\"Exited\", axis=1)\n",
    "y_train = train_df[\"Exited\"]\n",
    "X_test = test_df.drop(\"Exited\", axis=1)\n",
    "y_test = test_df[\"Exited\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize the split datasets\n",
    "vm_train_ds = vm.init_dataset(\n",
    "    input_id=\"train_dataset_final\",\n",
    "    dataset=train_df,\n",
    "    target_column=\"Exited\",\n",
    ")\n",
    "\n",
    "vm_test_ds = vm.init_dataset(\n",
    "    input_id=\"test_dataset_final\",\n",
    "    dataset=test_df,\n",
    "    target_column=\"Exited\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc2_4_'></a>\n",
    "\n",
    "### Import the champion model\n",
    "\n",
    "With our raw dataset assessed and preprocessed, let's go ahead and import the champion model submitted by the model development team in the format of a `.pkl` file: **[lr_model_champion.pkl](lr_model_champion.pkl)**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import the champion model\n",
    "import pickle as pkl\n",
    "\n",
    "with open(\"lr_model_champion.pkl\", \"rb\") as f:\n",
    "    log_reg = pkl.load(f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc2_5_'></a>\n",
    "\n",
    "### Train potential challenger model\n",
    "\n",
    "We'll also train our random forest classification challenger model to see how it compares:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import the Random Forest Classification model\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "\n",
    "# Create the model instance with 50 decision trees\n",
    "rf_model = RandomForestClassifier(\n",
    "    n_estimators=50,\n",
    "    random_state=42,\n",
    ")\n",
    "\n",
    "# Train the model\n",
    "rf_model.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc2_6_'></a>\n",
    "\n",
    "### Initialize the model objects\n",
    "\n",
    "In addition to the initialized datasets, you'll also need to initialize a ValidMind model object (`vm_model`) that can be passed to other functions for analysis and tests on the data for each of our two models:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize the champion logistic regression model\n",
    "vm_log_model = vm.init_model(\n",
    "    log_reg,\n",
    "    input_id=\"log_model_champion\",\n",
    ")\n",
    "\n",
    "# Initialize the challenger random forest classification model\n",
    "vm_rf_model = vm.init_model(\n",
    "    rf_model,\n",
    "    input_id=\"rf_model\",\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Assign predictions to Champion — Logistic regression model\n",
    "vm_train_ds.assign_predictions(model=vm_log_model)\n",
    "vm_test_ds.assign_predictions(model=vm_log_model)\n",
    "\n",
    "# Assign predictions to Challenger — Random forest classification model\n",
    "vm_train_ds.assign_predictions(model=vm_rf_model)\n",
    "vm_test_ds.assign_predictions(model=vm_rf_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc3_'></a>\n",
    "\n",
    "## Implementing custom tests\n",
    "\n",
    "Thanks to the model documentation ([Learn more ...](https://docs.validmind.ai/developer/validmind-library.html#for-model-development)), we know that the model development team implemented a custom test to further evaluate the performance of the champion model.\n",
    "\n",
    "In a usual model validation situation, you would load a saved custom test provided by the model development team. In the following section, we'll have you implement the same custom test and make it available for reuse, to familiarize you with the processes.\n",
    "\n",
    "<div class=\"alert alert-block alert-info\" style=\"background-color: #B5B5B510; color: black; border: 1px solid #083E44; border-left-width: 5px; box-shadow: 2px 2px 4px rgba(0, 0, 0, 0.2);border-radius: 5px;\"><span style=\"color: #083E44;\"><b>Want to learn more about custom tests?</b></span>\n",
    "<br></br>\n",
    "Refer to our in-depth introduction to custom tests: <a href=\"https://docs.validmind.ai/notebooks/code_samples/custom_tests/implement_custom_tests.html\" style=\"color: #DE257E;\"><b>Implement custom tests</b></a></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc3_1_'></a>\n",
    "\n",
    "### Implement a custom inline test\n",
    "\n",
    "Let's implement the same custom *inline test* that calculates the confusion matrix for a binary classification model that the model development team used in their performance evaluations.\n",
    "\n",
    "- An inline test refers to a test written and executed within the same environment as the code being tested — in this case, right in this Jupyter Notebook —  without requiring a separate test file or framework.\n",
    "- You'll note that the custom test function is just a regular Python function that can include and require any Python library as you see fit."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc3_1_1_'></a>\n",
    "\n",
    "#### Create a confusion matrix plot\n",
    "\n",
    "Let's first create a confusion matrix plot using the `confusion_matrix` function from the `sklearn.metrics` module:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "from sklearn import metrics\n",
    "\n",
    "# Get the predicted classes\n",
    "y_pred = log_reg.predict(vm_test_ds.x)\n",
    "\n",
    "confusion_matrix = metrics.confusion_matrix(y_test, y_pred)\n",
    "\n",
    "cm_display = metrics.ConfusionMatrixDisplay(\n",
    "    confusion_matrix=confusion_matrix, display_labels=[False, True]\n",
    ")\n",
    "cm_display.plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, create a [`@vm.test` wrapper](https://docs.validmind.ai/validmind/validmind.html#test) that will allow you to create a reusable test. **Note the following changes in the code below:**\n",
    "\n",
    "- The function `confusion_matrix` takes two arguments `dataset` and `model`. This is a `VMDataset` and `VMModel` object respectively.\n",
    "  - `VMDataset` objects allow you to access the dataset's true (target) values by accessing the `.y` attribute.\n",
    "  - `VMDataset` objects allow you to access the predictions for a given model by accessing the `.y_pred()` method.\n",
    "- The function docstring provides a description of what the test does. This will be displayed along with the result in this notebook as well as in the ValidMind Platform.\n",
    "- The function body calculates the confusion matrix using the `sklearn.metrics.confusion_matrix` function as we just did above.\n",
    "- The function then returns the `ConfusionMatrixDisplay.figure_` object — this is important as the ValidMind Library expects the output of the custom test to be a plot or a table.\n",
    "- The `@vm.test` decorator is doing the work of creating a wrapper around the function that will allow it to be run by the ValidMind Library. It also registers the test so it can be found by the ID `my_custom_tests.ConfusionMatrix`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "@vm.test(\"my_custom_tests.ConfusionMatrix\")\n",
    "def confusion_matrix(dataset, model):\n",
    "    \"\"\"The confusion matrix is a table that is often used to describe the performance of a classification model on a set of data for which the true values are known.\n",
    "\n",
    "    The confusion matrix is a 2x2 table that contains 4 values:\n",
    "\n",
    "    - True Positive (TP): the number of correct positive predictions\n",
    "    - True Negative (TN): the number of correct negative predictions\n",
    "    - False Positive (FP): the number of incorrect positive predictions\n",
    "    - False Negative (FN): the number of incorrect negative predictions\n",
    "\n",
    "    The confusion matrix can be used to assess the holistic performance of a classification model by showing the accuracy, precision, recall, and F1 score of the model on a single figure.\n",
    "    \"\"\"\n",
    "    y_true = dataset.y\n",
    "    y_pred = dataset.y_pred(model=model)\n",
    "\n",
    "    confusion_matrix = metrics.confusion_matrix(y_true, y_pred)\n",
    "\n",
    "    cm_display = metrics.ConfusionMatrixDisplay(\n",
    "        confusion_matrix=confusion_matrix, display_labels=[False, True]\n",
    "    )\n",
    "    cm_display.plot()\n",
    "\n",
    "    plt.close()  # close the plot to avoid displaying it\n",
    "\n",
    "    return cm_display.figure_  # return the figure object itself"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can now run the newly created custom test on both the training and test datasets for both models using the [`run_test()` function](https://docs.validmind.ai/validmind/validmind/tests.html#run_test):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Champion train and test\n",
    "vm.tests.run_test(\n",
    "    test_id=\"my_custom_tests.ConfusionMatrix:champion\",\n",
    "    input_grid={\n",
    "        \"dataset\": [vm_train_ds,vm_test_ds],\n",
    "        \"model\" : [vm_log_model]\n",
    "    }\n",
    ").log()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Challenger train and test\n",
    "vm.tests.run_test(\n",
    "    test_id=\"my_custom_tests.ConfusionMatrix:challenger\",\n",
    "    input_grid={\n",
    "        \"dataset\": [vm_train_ds,vm_test_ds],\n",
    "        \"model\" : [vm_rf_model]\n",
    "    }\n",
    ").log()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\" style=\"background-color: #B5B5B510; color: black; border: 1px solid #083E44; border-left-width: 5px; box-shadow: 2px 2px 4px rgba(0, 0, 0, 0.2);border-radius: 5px;\"><span style=\"color: #083E44;\"><b>Note the output returned indicating that a test-driven block doesn't currently exist in your model's documentation for some test IDs. </b></span>\n",
    "<br></br>\n",
    "That's expected, as when we run validations tests the results logged need to be manually added to your report as part of your compliance assessment process within the ValidMind Platform.</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc3_1_2_'></a>\n",
    "\n",
    "#### Add parameters to custom tests\n",
    "\n",
    "Custom tests can take parameters just like any other function. To demonstrate, let's modify the `confusion_matrix` function to take an additional parameter `normalize` that will allow you to normalize the confusion matrix:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "@vm.test(\"my_custom_tests.ConfusionMatrix\")\n",
    "def confusion_matrix(dataset, model, normalize=False):\n",
    "    \"\"\"The confusion matrix is a table that is often used to describe the performance of a classification model on a set of data for which the true values are known.\n",
    "\n",
    "    The confusion matrix is a 2x2 table that contains 4 values:\n",
    "\n",
    "    - True Positive (TP): the number of correct positive predictions\n",
    "    - True Negative (TN): the number of correct negative predictions\n",
    "    - False Positive (FP): the number of incorrect positive predictions\n",
    "    - False Negative (FN): the number of incorrect negative predictions\n",
    "\n",
    "    The confusion matrix can be used to assess the holistic performance of a classification model by showing the accuracy, precision, recall, and F1 score of the model on a single figure.\n",
    "    \"\"\"\n",
    "    y_true = dataset.y\n",
    "    y_pred = dataset.y_pred(model=model)\n",
    "\n",
    "    if normalize:\n",
    "        confusion_matrix = metrics.confusion_matrix(y_true, y_pred, normalize=\"all\")\n",
    "    else:\n",
    "        confusion_matrix = metrics.confusion_matrix(y_true, y_pred)\n",
    "\n",
    "    cm_display = metrics.ConfusionMatrixDisplay(\n",
    "        confusion_matrix=confusion_matrix, display_labels=[False, True]\n",
    "    )\n",
    "    cm_display.plot()\n",
    "\n",
    "    plt.close()  # close the plot to avoid displaying it\n",
    "\n",
    "    return cm_display.figure_  # return the figure object itself"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc3_1_3_'></a>\n",
    "\n",
    "#### Pass parameters to custom tests\n",
    "\n",
    "You can pass parameters to custom tests by providing a dictionary of parameters to the `run_test()` function.\n",
    "\n",
    "- The parameters will override any default parameters set in the custom test definition. Note that `dataset` and `model` are still passed as `inputs`.\n",
    "- Since these are `VMDataset` or `VMModel` inputs, they have a special meaning.\n",
    "\n",
    "Re-running and logging the custom confusion matrix with `normalize=True` for both models and our testing dataset looks like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Champion with test dataset and normalize=True\n",
    "vm.tests.run_test(\n",
    "    test_id=\"my_custom_tests.ConfusionMatrix:test_normalized_champion\",\n",
    "    input_grid={\n",
    "        \"dataset\": [vm_test_ds],\n",
    "        \"model\" : [vm_log_model]\n",
    "    },\n",
    "    params={\"normalize\": True}\n",
    ").log()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Challenger with test dataset and normalize=True\n",
    "vm.tests.run_test(\n",
    "    test_id=\"my_custom_tests.ConfusionMatrix:test_normalized_challenger\",\n",
    "    input_grid={\n",
    "        \"dataset\": [vm_test_ds],\n",
    "        \"model\" : [vm_rf_model]\n",
    "    },\n",
    "    params={\"normalize\": True}\n",
    ").log()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc3_2_'></a>\n",
    "\n",
    "### Use external test providers\n",
    "\n",
    "Sometimes you may want to reuse the same set of custom tests across multiple models and share them with others in your organization, like the model development team would have done with you in this example workflow featured in this series of notebooks. In this case, you can create an external custom *test provider* that will allow you to load custom tests from a local folder or a Git repository.\n",
    "\n",
    "In this section you will learn how to declare a local filesystem test provider that allows loading tests from a local folder following these high level steps:\n",
    "\n",
    "1. Create a folder of custom tests from existing inline tests (tests that exist in your active Jupyter Notebook)\n",
    "2. Save an inline test to a file\n",
    "3. Define and register a [`LocalTestProvider`](https://docs.validmind.ai/validmind/validmind/tests.html#LocalTestProvider) that points to that folder\n",
    "4. Run test provider tests\n",
    "5. Add the test results to your documentation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc3_2_1_'></a>\n",
    "\n",
    "#### Create custom tests folder\n",
    "\n",
    "Let's start by creating a new folder that will contain reusable custom tests from your existing inline tests.\n",
    "\n",
    "The following code snippet will create a new `my_tests` directory in the current working directory if it doesn't exist:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tests_folder = \"my_tests\"\n",
    "\n",
    "import os\n",
    "\n",
    "# create tests folder\n",
    "os.makedirs(tests_folder, exist_ok=True)\n",
    "\n",
    "# remove existing tests\n",
    "for f in os.listdir(tests_folder):\n",
    "    # remove files and pycache\n",
    "    if f.endswith(\".py\") or f == \"__pycache__\":\n",
    "        os.system(f\"rm -rf {tests_folder}/{f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After running the command above, confirm that a new `my_tests` directory was created successfully. For example:\n",
    "\n",
    "```\n",
    "~/notebooks/tutorials/model_validation/my_tests/\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc3_2_2_'></a>\n",
    "\n",
    "#### Save an inline test\n",
    "\n",
    "The `@vm.test` decorator we used in **Implement a custom inline test** above to register one-off custom tests also includes a convenience method on the function object that allows you to simply call `<func_name>.save()` to save the test to a Python file at a specified path.\n",
    "\n",
    "While `save()` will get you started by creating the file and saving the function code with the correct name, it won't automatically include any imports, or other functions or variables, outside of the functions that are needed for the test to run. To solve this, pass in an optional `imports` argument ensuring necessary imports are added to the file.\n",
    "\n",
    "The `confusion_matrix` test requires the following additional imports:\n",
    "\n",
    "```python\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn import metrics\n",
    "```\n",
    "\n",
    "Let's pass these imports to the `save()` method to ensure they are included in the file with the following command:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "confusion_matrix.save(\n",
    "    # Save it to the custom tests folder we created\n",
    "    tests_folder,\n",
    "    imports=[\"import matplotlib.pyplot as plt\", \"from sklearn import metrics\"],\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- [x] Confirm that the `save()` method saved the `confusion_matrix` function to a file named `ConfusionMatrix.py` in the `my_tests` folder.\n",
    "- [x] Note that the new file provides some context on the origin of the test, which is useful for traceability:\n",
    "\n",
    "    ```\n",
    "    # Saved from __main__.confusion_matrix\n",
    "    # Original Test ID: my_custom_tests.ConfusionMatrix\n",
    "    # New Test ID: <test_provider_namespace>.ConfusionMatrix\n",
    "    ```\n",
    "\n",
    "- [x]  Additionally, the new test function has been stripped off its decorator, as it now resides in a file that will be loaded by the test provider:\n",
    "\n",
    "    ```python\n",
    "    def ConfusionMatrix(dataset, model, normalize=False):\n",
    "    ```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc3_2_3_'></a>\n",
    "\n",
    "#### Register a local test provider\n",
    "\n",
    "Now that your `my_tests` folder has a sample custom test, let's initialize a test provider that will tell the ValidMind Library where to find your custom tests:\n",
    "\n",
    "- ValidMind offers out-of-the-box test providers for local tests (tests in a folder) or a Github provider for tests in a Github repository.\n",
    "- You can also create your own test provider by creating a class that has a [`load_test` method](https://docs.validmind.ai/validmind/validmind/tests.html#load_test) that takes a test ID and returns the test function matching that ID.\n",
    "\n",
    "<div class=\"alert alert-block alert-info\" style=\"background-color: #B5B5B510; color: black; border: 1px solid #083E44; border-left-width: 5px; box-shadow: 2px 2px 4px rgba(0, 0, 0, 0.2);border-radius: 5px;\"><span style=\"color: #083E44;\"><b>Want to learn more about test providers?</b></span>\n",
    "<br></br>\n",
    "An extended introduction to test providers can be found in: <a href=\"https://docs.validmind.ai/notebooks/code_samples/custom_tests/integrate_external_test_providers.html\" style=\"color: #DE257E;\"><b>Integrate external test providers</b></a></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Initialize a local test provider\n",
    "\n",
    "For most use cases, using a `LocalTestProvider` that allows you to load custom tests from a designated directory should be sufficient.\n",
    "\n",
    "**The most important attribute for a test provider is its `namespace`.** This is a string that will be used to prefix test IDs in model documentation. This allows you to have multiple test providers with tests that can even share the same ID, but are distinguished by their namespace.\n",
    "\n",
    "Let's go ahead and load the custom tests from our `my_tests` directory:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from validmind.tests import LocalTestProvider\n",
    "\n",
    "# initialize the test provider with the tests folder we created earlier\n",
    "my_test_provider = LocalTestProvider(tests_folder)\n",
    "\n",
    "vm.tests.register_test_provider(\n",
    "    namespace=\"my_test_provider\",\n",
    "    test_provider=my_test_provider,\n",
    ")\n",
    "# `my_test_provider.load_test()` will be called for any test ID that starts with `my_test_provider`\n",
    "# e.g. `my_test_provider.ConfusionMatrix` will look for a function named `ConfusionMatrix` in `my_tests/ConfusionMatrix.py` file"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Run test provider tests\n",
    "\n",
    "Now that we've set up the test provider, we can run any test that's located in the tests folder by using the `run_test()` method as with any other test:\n",
    "\n",
    "- For tests that reside in a test provider directory, the test ID will be the `namespace` specified when registering the provider, followed by the path to the test file relative to the tests folder.\n",
    "- For example, the Confusion Matrix test we created earlier will have the test ID `my_test_provider.ConfusionMatrix`. You could organize the tests in subfolders, say `classification` and `regression`, and the test ID for the Confusion Matrix test would then be `my_test_provider.classification.ConfusionMatrix`.\n",
    "\n",
    "Let's go ahead and re-run the confusion matrix test with our testing dataset for our two models by using the test ID `my_test_provider.ConfusionMatrix`. This should load the test from the test provider and run it as before."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Champion with test dataset and test provider custom test\n",
    "vm.tests.run_test(\n",
    "    test_id=\"my_test_provider.ConfusionMatrix:champion\",\n",
    "    input_grid={\n",
    "        \"dataset\": [vm_test_ds],\n",
    "        \"model\" : [vm_log_model]\n",
    "    }\n",
    ").log()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Challenger with test dataset  and test provider custom test\n",
    "vm.tests.run_test(\n",
    "    test_id=\"my_test_provider.ConfusionMatrix:challenger\",\n",
    "    input_grid={\n",
    "        \"dataset\": [vm_test_ds],\n",
    "        \"model\" : [vm_rf_model]\n",
    "    }\n",
    ").log()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc4_'></a>\n",
    "\n",
    "## Verify test runs\n",
    "\n",
    "Our final task is to verify that all the tests provided by the model development team were run and reported accurately. Note the appended `result_ids` to delineate which dataset we ran the test with for the relevant tests.\n",
    "\n",
    "Here, we'll specify all the tests we'd like to independently rerun in a dictionary called `test_config`. **Note here that `inputs` and `input_grid` expect the `input_id` of the dataset or model as the value rather than the variable name we specified**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_config = {\n",
    "    # Run with the raw dataset\n",
    "    'validmind.data_validation.DatasetDescription:raw_data': {\n",
    "        'inputs': {'dataset': 'raw_dataset'}\n",
    "    },\n",
    "    'validmind.data_validation.DescriptiveStatistics:raw_data': {\n",
    "        'inputs': {'dataset': 'raw_dataset'}\n",
    "    },\n",
    "    'validmind.data_validation.MissingValues:raw_data': {\n",
    "        'inputs': {'dataset': 'raw_dataset'},\n",
    "        'params': {'min_threshold': 1}\n",
    "    },\n",
    "    'validmind.data_validation.ClassImbalance:raw_data': {\n",
    "        'inputs': {'dataset': 'raw_dataset'},\n",
    "        'params': {'min_percent_threshold': 10}\n",
    "    },\n",
    "    'validmind.data_validation.Duplicates:raw_data': {\n",
    "        'inputs': {'dataset': 'raw_dataset'},\n",
    "        'params': {'min_threshold': 1}\n",
    "    },\n",
    "    'validmind.data_validation.HighCardinality:raw_data': {\n",
    "        'inputs': {'dataset': 'raw_dataset'},\n",
    "        'params': {\n",
    "            'num_threshold': 100,\n",
    "            'percent_threshold': 0.1,\n",
    "            'threshold_type': 'percent'\n",
    "        }\n",
    "    },\n",
    "    'validmind.data_validation.Skewness:raw_data': {\n",
    "        'inputs': {'dataset': 'raw_dataset'},\n",
    "        'params': {'max_threshold': 1}\n",
    "    },\n",
    "    'validmind.data_validation.UniqueRows:raw_data': {\n",
    "        'inputs': {'dataset': 'raw_dataset'},\n",
    "        'params': {'min_percent_threshold': 1}\n",
    "    },\n",
    "    'validmind.data_validation.TooManyZeroValues:raw_data': {\n",
    "        'inputs': {'dataset': 'raw_dataset'},\n",
    "        'params': {'max_percent_threshold': 0.03}\n",
    "    },\n",
    "    'validmind.data_validation.IQROutliersTable:raw_data': {\n",
    "        'inputs': {'dataset': 'raw_dataset'},\n",
    "        'params': {'threshold': 5}\n",
    "    },\n",
    "    # Run with the preprocessed dataset\n",
    "    'validmind.data_validation.DescriptiveStatistics:preprocessed_data': {\n",
    "        'inputs': {'dataset': 'raw_dataset_preprocessed'}\n",
    "    },\n",
    "    'validmind.data_validation.TabularDescriptionTables:preprocessed_data': {\n",
    "        'inputs': {'dataset': 'raw_dataset_preprocessed'}\n",
    "    },\n",
    "    'validmind.data_validation.MissingValues:preprocessed_data': {\n",
    "        'inputs': {'dataset': 'raw_dataset_preprocessed'},\n",
    "        'params': {'min_threshold': 1}\n",
    "    },\n",
    "    'validmind.data_validation.TabularNumericalHistograms:preprocessed_data': {\n",
    "        'inputs': {'dataset': 'raw_dataset_preprocessed'}\n",
    "    },\n",
    "    'validmind.data_validation.TabularCategoricalBarPlots:preprocessed_data': {\n",
    "        'inputs': {'dataset': 'raw_dataset_preprocessed'}\n",
    "    },\n",
    "    'validmind.data_validation.TargetRateBarPlots:preprocessed_data': {\n",
    "        'inputs': {'dataset': 'raw_dataset_preprocessed'},\n",
    "        'params': {'default_column': 'loan_status'}\n",
    "    },\n",
    "    # Run with the training and test datasets\n",
    "    'validmind.data_validation.DescriptiveStatistics:development_data': {\n",
    "        'input_grid': {'dataset': ['train_dataset_final', 'test_dataset_final']}\n",
    "    },\n",
    "    'validmind.data_validation.TabularDescriptionTables:development_data': {\n",
    "        'input_grid': {'dataset': ['train_dataset_final', 'test_dataset_final']}\n",
    "    },\n",
    "    'validmind.data_validation.ClassImbalance:development_data': {\n",
    "        'input_grid': {'dataset': ['train_dataset_final', 'test_dataset_final']},\n",
    "        'params': {'min_percent_threshold': 10}\n",
    "    },\n",
    "    'validmind.data_validation.UniqueRows:development_data': {\n",
    "        'input_grid': {'dataset': ['train_dataset_final', 'test_dataset_final']},\n",
    "        'params': {'min_percent_threshold': 1}\n",
    "    },\n",
    "    'validmind.data_validation.TabularNumericalHistograms:development_data': {\n",
    "        'input_grid': {'dataset': ['train_dataset_final', 'test_dataset_final']}\n",
    "    },\n",
    "    'validmind.data_validation.MutualInformation:development_data': {\n",
    "        'input_grid': {'dataset': ['train_dataset_final', 'test_dataset_final']},\n",
    "        'params': {'min_threshold': 0.01}\n",
    "    },\n",
    "    'validmind.data_validation.PearsonCorrelationMatrix:development_data': {\n",
    "        'input_grid': {'dataset': ['train_dataset_final', 'test_dataset_final']}\n",
    "    },\n",
    "    'validmind.data_validation.HighPearsonCorrelation:development_data': {\n",
    "        'input_grid': {'dataset': ['train_dataset_final', 'test_dataset_final']},\n",
    "        'params': {'max_threshold': 0.3, 'top_n_correlations': 10}\n",
    "    },\n",
    "    'validmind.model_validation.ModelMetadata': {\n",
    "        'input_grid': {'model': ['log_model_champion', 'rf_model']}\n",
    "    },\n",
    "    'validmind.model_validation.sklearn.ModelParameters': {\n",
    "        'input_grid': {'model': ['log_model_champion', 'rf_model']}\n",
    "    },\n",
    "    'validmind.model_validation.sklearn.ROCCurve': {\n",
    "        'input_grid': {'dataset': ['train_dataset_final', 'test_dataset_final'], 'model': ['log_model_champion']}\n",
    "    },\n",
    "    'validmind.model_validation.sklearn.MinimumROCAUCScore': {\n",
    "        'input_grid': {'dataset': ['train_dataset_final', 'test_dataset_final'], 'model': ['log_model_champion']},\n",
    "        'params': {'min_threshold': 0.5}\n",
    "    }\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then batch run and log our tests in `test_config`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for t in test_config:\n",
    "    print(t)\n",
    "    try:\n",
    "        # Check if test has input_grid\n",
    "        if 'input_grid' in test_config[t]:\n",
    "            # For tests with input_grid, pass the input_grid configuration\n",
    "            if 'params' in test_config[t]:\n",
    "                vm.tests.run_test(t, input_grid=test_config[t]['input_grid'], params=test_config[t]['params']).log()\n",
    "            else:\n",
    "                vm.tests.run_test(t, input_grid=test_config[t]['input_grid']).log()\n",
    "        else:\n",
    "            # Original logic for regular inputs\n",
    "            if 'params' in test_config[t]:\n",
    "                vm.tests.run_test(t, inputs=test_config[t]['inputs'], params=test_config[t]['params']).log()\n",
    "            else:\n",
    "                vm.tests.run_test(t, inputs=test_config[t]['inputs']).log()\n",
    "    except Exception as e:\n",
    "        print(f\"Error running test {t}: {str(e)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc5_'></a>\n",
    "\n",
    "## In summary\n",
    "\n",
    "In this final notebook, you learned how to:\n",
    "\n",
    "- [x] Implement a custom inline test\n",
    "- [x] Run and log your custom inline tests\n",
    "- [x] Use external custom test providers\n",
    "- [x] Run and log tests from your custom test providers\n",
    "- [x] Re-run tests provided by your model development team to verify that they were run and reported accurately\n",
    "\n",
    "With our ValidMind for model validation series of notebooks, you learned how to validate a model end-to-end with the ValidMind Library by running through some common scenarios in a typical model validation setting:\n",
    "\n",
    "- Verifying the data quality steps performed by the model development team\n",
    "- Independently replicating the champion model's results and conducting additional tests to assess performance, stability, and robustness\n",
    "- Setting up test inputs and a challenger model for comparative analysis\n",
    "- Running validation tests, analyzing results, and logging findings to ValidMind"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc6_'></a>\n",
    "\n",
    "## Next steps"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc6_1_'></a>\n",
    "\n",
    "### Work with your validation report\n",
    "\n",
    "Now that you've logged all your test results and verified the work done by the model development team, head to the ValidMind Platform to wrap up your validation report. Continue to work on your validation report by:\n",
    "\n",
    "- **Inserting additional test results:** Click **Link Evidence to Report** under any section of 2. Validation in your validation report. (Learn more: [Link evidence to reports](https://docs.validmind.ai/guide/model-validation/assess-compliance.html#link-evidence-to-reports))\n",
    "\n",
    "- **Making qualitative edits to your test descriptions:** Expand any linked evidence under Validator Evidence and click **See evidence details** to review and edit the ValidMind-generated test descriptions for quality and accuracy. (Learn more: [Preparing validation reports](https://docs.validmind.ai/guide/model-validation/preparing-validation-reports.html#get-started))\n",
    "\n",
    "- **Adding more findings:** Click **Link Finding to Report** in any validation report section, then click **+ Create New Finding**. (Learn more: [Add and manage model findings](https://docs.validmind.ai/guide/model-validation/add-manage-model-findings.html))\n",
    "\n",
    "- **Adding risk assessment notes:** Click under **Risk Assessment Notes** in any validation report section to access the text editor and content editing toolbar, including an option to generate a draft with AI. Once generated, edit your ValidMind-generated test descriptions to adhere to your organization's requirements. (Learn more: [Work with content blocks](https://docs.validmind.ai/guide/model-documentation/work-with-content-blocks.html#content-editing-toolbar))\n",
    "\n",
    "- **Assessing compliance:** Under the Guideline for any validation report section, click **ASSESSMENT** and select the compliance status from the drop-down menu. (Learn more: [Provide compliance assessments](https://docs.validmind.ai/guide/model-validation/assess-compliance.html#provide-compliance-assessments))\n",
    "\n",
    "- **Collaborate with other stakeholders:** Use the ValidMind Platform's real-time collaborative features to work seamlessly together with the rest of your organization, including model developers. Propose suggested changes in the model documentation, work with versioned history, and use comments to discuss specific portions of the model documentation. (Learn more: [Collaborate with others](https://docs.validmind.ai/guide/model-documentation/collaborate-with-others.html))\n",
    "\n",
    "When your validation report is complete and ready for review, submit it for approval from the same ValidMind Platform where you made your edits and collaborated with the rest of your organization, ensuring transparency and a thorough model validation history. (Learn more: [Submit for approval](https://docs.validmind.ai/guide/model-documentation/submit-for-approval.html))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc6_2_'></a>\n",
    "\n",
    "### Learn more\n",
    "\n",
    "Now that you're familiar with the basics, you can explore the following notebooks to get a deeper understanding on how the ValidMind Library assists you in streamlining model validation:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc6_2_1_'></a>\n",
    "\n",
    "#### More how-to guides and code samples\n",
    "\n",
    "- [Explore available tests in detail](../../how_to/explore_tests.ipynb)\n",
    "- [In-depth guide on running dataset based tests](../../how_to/run_tests/1_run_dataset_based_tests.ipynb)\n",
    "- [In-depth guide for running comparison tests](../../how_to/run_tests/2_run_comparison_tests.ipynb)\n",
    "- [In-depth guide for implementing custom tests](../../code_samples/custom_tests/implement_custom_tests.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc6_2_2_'></a>\n",
    "\n",
    "#### Discover more learning resources\n",
    "\n",
    "All notebook samples can be found in the following directories of the ValidMind Library GitHub repository:\n",
    "\n",
    "- [Code samples](https://github.com/validmind/validmind-library/tree/main/notebooks/code_samples)\n",
    "- [How-to guides](https://github.com/validmind/validmind-library/tree/main/notebooks/how_to)\n",
    "\n",
    "Or, visit our [documentation](https://docs.validmind.ai/) to learn more about ValidMind."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "ValidMind Library",
   "language": "python",
   "name": "validmind"
  },
  "language_info": {
   "name": "python",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
