{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# ValidMind for model development 4 — Finalize testing and documentation\n",
    "\n",
    "Learn how to use ValidMind for your end-to-end model documentation process with our introductory notebook series. In this last notebook, finalize the testing and documentation of your model and have a fully documented sample model ready for review.\n",
    "\n",
    "We'll first use [`run_documentation_tests()`](https://docs.validmind.ai/validmind/validmind.html#run_documentation_tests) previously covered in **[2 — Start the model development process](2-start_development_process.ipynb)** to ensure that your custom test results generated in **[3 — Integrate custom tests](3-integrate_custom_tests.ipynb)** are included in your documentation. Then, we'll view and update the configuration for the entire model documentation template to suit your needs.\n",
    "\n",
    "<div class=\"alert alert-block alert-info\" style=\"background-color: #B5B5B510; color: black; border: 1px solid #083E44; border-left-width: 5px; box-shadow: 2px 2px 4px rgba(0, 0, 0, 0.2);border-radius: 5px;\"><span style=\"color: #083E44;\"><b>Learn by doing</b></span>\n",
    "<br></br>\n",
    "Our course tailor-made for developers new to ValidMind combines this series of notebooks with more a more in-depth introduction to the ValidMind Platform — <a href=\"https://docs.validmind.ai/training/developer-fundamentals/developer-fundamentals-register.html\" style=\"color: #DE257E;\"><b>Developer Fundamentals</b></a></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "::: {.content-hidden when-format=\"html\"}\n",
    "## Contents    \n",
    "- [Prerequisites](#toc1_)    \n",
    "- [Setting up](#toc2_)    \n",
    "  - [Initialize the ValidMind Library](#toc2_1_)    \n",
    "  - [Import sample dataset](#toc2_2_)    \n",
    "    - [Remove highly correlated features](#toc2_2_1_)    \n",
    "  - [Train the model](#toc2_3_)    \n",
    "    - [Initialize the ValidMind objects](#toc2_3_1_)    \n",
    "    - [Assign predictions](#toc2_3_2_)    \n",
    "  - [Add custom tests](#toc2_4_)    \n",
    "    - [Implement custom inline test](#toc2_4_1_)    \n",
    "    - [Add a local test provider](#toc2_4_2_)    \n",
    "- [Reconnect to ValidMind](#toc3_)    \n",
    "- [Include custom test results](#toc4_)    \n",
    "- [Documentation template configuration](#toc5_)    \n",
    "  - [Update the config](#toc5_1_)    \n",
    "- [In summary](#toc6_)    \n",
    "- [Next steps](#toc7_)    \n",
    "  - [Work with your model documentation](#toc7_1_)    \n",
    "  - [Learn more](#toc7_2_)    \n",
    "    - [Use cases](#toc7_2_1_)    \n",
    "    - [More how-to guides and code samples](#toc7_2_2_)    \n",
    "    - [Discover more learning resources](#toc7_2_3_)    \n",
    "\n",
    ":::\n",
    "<!-- jn-toc-notebook-config\n",
    "\tnumbering=false\n",
    "\tanchor=true\n",
    "\tflat=false\n",
    "\tminLevel=2\n",
    "\tmaxLevel=4\n",
    "\t/jn-toc-notebook-config -->\n",
    "<!-- THIS CELL WILL BE REPLACED ON TOC UPDATE. DO NOT WRITE YOUR TEXT IN THIS CELL -->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc1_'></a>\n",
    "\n",
    "## Prerequisites\n",
    "\n",
    "In order to finalize the testing and documentation for your sample model, you'll need to first have:\n",
    "\n",
    "- [x] Registered a model within the ValidMind Platform with a predefined documentation template\n",
    "- [x] Installed the ValidMind Library in your local environment, allowing you to access all its features\n",
    "- [x] Learned how to import and initialize datasets for use with ValidMind\n",
    "- [x] Learned how to run and log default and custom tests with ValidMind, including from external test providers\n",
    "- [x] Inserted test-driven blocks for the results of the following tests into your model's documentation:\n",
    "    - [x] `HighPearsonCorrelation:balanced_raw_dataset`\n",
    "    - [x] `my_test_provider.ConfusionMatrix`\n",
    "    - [x] `my_custom_tests.ConfusionMatrix:test_dataset_normalized`\n",
    "\n",
    "<div class=\"alert alert-block alert-info\" style=\"background-color: #B5B5B510; color: black; border: 1px solid #083E44; border-left-width: 5px; box-shadow: 2px 2px 4px rgba(0, 0, 0, 0.2);border-radius: 5px;\"><span style=\"color: #083E44;\"><b>Need help with the above steps?</b></span>\n",
    "<br></br>\n",
    "Refer to the first three notebooks in this series:\n",
    "\n",
    "- <a href=\"1-set_up_validmind.ipynb\" style=\"color: #DE257E;\"><b>1 — Set up the ValidMind Library</b></a>\n",
    "- <a href=\"2-start_development_process.ipynb\" style=\"color: #DE257E;\"><b>2 — Start the model development process</b></a>\n",
    "- <a href=\"3-integrate_custom_tests.ipynb\" style=\"color: #DE257E;\"><b>3 — Integrate custom tests</b></a>\n",
    "\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc2_'></a>\n",
    "\n",
    "## Setting up\n",
    "\n",
    "This section should be very familiar to you now — as we performed the same actions in the previous two notebooks in this series."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc2_1_'></a>\n",
    "\n",
    "### Initialize the ValidMind Library\n",
    "\n",
    "As usual, let's first connect up the ValidMind Library to our model we previously registered in the ValidMind Platform:\n",
    "\n",
    "1. In a browser, [log in to ValidMind](https://docs.validmind.ai/guide/configuration/log-in-to-validmind.html).\n",
    "\n",
    "2. In the left sidebar, navigate to **Inventory** and select the model you registered for this \"ValidMind for model development\" series of notebooks.\n",
    "\n",
    "3. Go to **Getting Started** and click **Copy snippet to clipboard**.\n",
    "\n",
    "Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Make sure the ValidMind Library is installed\n",
    "\n",
    "%pip install -q validmind\n",
    "\n",
    "# Load your model identifier credentials from an `.env` file\n",
    "\n",
    "%load_ext dotenv\n",
    "%dotenv .env\n",
    "\n",
    "# Or replace with your code snippet\n",
    "\n",
    "import validmind as vm\n",
    "\n",
    "vm.init(\n",
    "    # api_host=\"...\",\n",
    "    # api_key=\"...\",\n",
    "    # api_secret=\"...\",\n",
    "    # model=\"...\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc2_2_'></a>\n",
    "\n",
    "### Import sample dataset\n",
    "\n",
    "Next, we'll import the same public [Bank Customer Churn Prediction](https://www.kaggle.com/datasets/shantanudhakadd/bank-customer-churn-prediction) dataset from Kaggle we used in the last notebooks so that we have something to work with:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from validmind.datasets.classification import customer_churn as demo_dataset\n",
    "\n",
    "print(\n",
    "    f\"Loaded demo dataset with: \\n\\n\\t• Target column: '{demo_dataset.target_column}' \\n\\t• Class labels: {demo_dataset.class_labels}\"\n",
    ")\n",
    "\n",
    "raw_df = demo_dataset.load_data()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll apply a simple rebalancing technique to the dataset before continuing:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "raw_copy_df = raw_df.sample(frac=1)  # Create a copy of the raw dataset\n",
    "\n",
    "# Create a balanced dataset with the same number of exited and not exited customers\n",
    "exited_df = raw_copy_df.loc[raw_copy_df[\"Exited\"] == 1]\n",
    "not_exited_df = raw_copy_df.loc[raw_copy_df[\"Exited\"] == 0].sample(n=exited_df.shape[0])\n",
    "\n",
    "balanced_raw_df = pd.concat([exited_df, not_exited_df])\n",
    "balanced_raw_df = balanced_raw_df.sample(frac=1, random_state=42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc2_2_1_'></a>\n",
    "\n",
    "#### Remove highly correlated features\n",
    "\n",
    "Let's also quickly remove highly correlated features from the dataset using the output from a ValidMind test.\n",
    "\n",
    "As you learned previously, before we can run tests you'll need to initialize a ValidMind dataset object:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Register new data and now 'balanced_raw_dataset' is the new dataset object of interest\n",
    "vm_balanced_raw_dataset = vm.init_dataset(\n",
    "    dataset=balanced_raw_df,\n",
    "    input_id=\"balanced_raw_dataset\",\n",
    "    target_column=\"Exited\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With our balanced dataset initialized, we can then run our test and utilize the output to help us identify the features we want to remove:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run HighPearsonCorrelation test with our balanced dataset as input and return a result object\n",
    "corr_result = vm.tests.run_test(\n",
    "    test_id=\"validmind.data_validation.HighPearsonCorrelation\",\n",
    "    params={\"max_threshold\": 0.3},\n",
    "    inputs={\"dataset\": vm_balanced_raw_dataset},\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# From result object, extract table from `corr_result.tables`\n",
    "features_df = corr_result.tables[0].data\n",
    "features_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Extract list of features that failed the test\n",
    "high_correlation_features = features_df[features_df[\"Pass/Fail\"] == \"Fail\"][\"Columns\"].tolist()\n",
    "high_correlation_features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Extract feature names from the list of strings\n",
    "high_correlation_features = [feature.split(\",\")[0].strip(\"()\") for feature in high_correlation_features]\n",
    "high_correlation_features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can then re-initialize the dataset with a different `input_id` and the highly correlated features removed and re-run the test for confirmation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Remove the highly correlated features from the dataset\n",
    "balanced_raw_no_age_df = balanced_raw_df.drop(columns=high_correlation_features)\n",
    "\n",
    "# Re-initialize the dataset object\n",
    "vm_raw_dataset_preprocessed = vm.init_dataset(\n",
    "    dataset=balanced_raw_no_age_df,\n",
    "    input_id=\"raw_dataset_preprocessed\",\n",
    "    target_column=\"Exited\",\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Re-run the test with the reduced feature set\n",
    "corr_result = vm.tests.run_test(\n",
    "    test_id=\"validmind.data_validation.HighPearsonCorrelation\",\n",
    "    params={\"max_threshold\": 0.3},\n",
    "    inputs={\"dataset\": vm_raw_dataset_preprocessed},\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc2_3_'></a>\n",
    "\n",
    "### Train the model\n",
    "\n",
    "We'll then use ValidMind tests to train a simple logistic regression model on our prepared dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# First encode the categorical features in our dataset with the highly correlated features removed\n",
    "balanced_raw_no_age_df = pd.get_dummies(\n",
    "    balanced_raw_no_age_df, columns=[\"Geography\", \"Gender\"], drop_first=True\n",
    ")\n",
    "balanced_raw_no_age_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Split the processed dataset into train and test\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "train_df, test_df = train_test_split(balanced_raw_no_age_df, test_size=0.20)\n",
    "\n",
    "X_train = train_df.drop(\"Exited\", axis=1)\n",
    "y_train = train_df[\"Exited\"]\n",
    "X_test = test_df.drop(\"Exited\", axis=1)\n",
    "y_test = test_df[\"Exited\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "# Logistic Regression grid params\n",
    "log_reg_params = {\n",
    "    \"penalty\": [\"l1\", \"l2\"],\n",
    "    \"C\": [0.001, 0.01, 0.1, 1, 10, 100, 1000],\n",
    "    \"solver\": [\"liblinear\"],\n",
    "}\n",
    "\n",
    "# Grid search for Logistic Regression\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "grid_log_reg = GridSearchCV(LogisticRegression(), log_reg_params)\n",
    "grid_log_reg.fit(X_train, y_train)\n",
    "\n",
    "# Logistic Regression best estimator\n",
    "log_reg = grid_log_reg.best_estimator_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc2_3_1_'></a>\n",
    "\n",
    "#### Initialize the ValidMind objects\n",
    "\n",
    "Let's initialize the ValidMind `Dataset` and `Model` objects in preparation for assigning model predictions to each dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize the datasets into their own dataset objects\n",
    "vm_train_ds = vm.init_dataset(\n",
    "    input_id=\"train_dataset_final\",\n",
    "    dataset=train_df,\n",
    "    target_column=\"Exited\",\n",
    ")\n",
    "\n",
    "vm_test_ds = vm.init_dataset(\n",
    "    input_id=\"test_dataset_final\",\n",
    "    dataset=test_df,\n",
    "    target_column=\"Exited\",\n",
    ")\n",
    "\n",
    "# Initialize a model object\n",
    "vm_model = vm.init_model(log_reg, input_id=\"log_reg_model_v1\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc2_3_2_'></a>\n",
    "\n",
    "#### Assign predictions\n",
    "\n",
    "Once the model is registered, we'll assign predictions to the training and test datasets:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vm_train_ds.assign_predictions(model=vm_model)\n",
    "vm_test_ds.assign_predictions(model=vm_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc2_4_'></a>\n",
    "\n",
    "### Add custom tests\n",
    "\n",
    "We'll also add the same custom tests we implemented in the previous notebook so that this session has access to the same custom inline test and local test provider."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc2_4_1_'></a>\n",
    "\n",
    "#### Implement custom inline test\n",
    "\n",
    "Let's set up a custom inline test that calculates the confusion matrix for a binary classification model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# First create a confusion matrix plot\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn import metrics\n",
    "\n",
    "# Get the predicted classes\n",
    "y_pred = log_reg.predict(vm_test_ds.x)\n",
    "\n",
    "confusion_matrix = metrics.confusion_matrix(y_test, y_pred)\n",
    "\n",
    "cm_display = metrics.ConfusionMatrixDisplay(\n",
    "    confusion_matrix=confusion_matrix, display_labels=[False, True]\n",
    ")\n",
    "cm_display.plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the reusable ConfusionMatrix inline test with normalized matrix\n",
    "@vm.test(\"my_custom_tests.ConfusionMatrix\")\n",
    "def confusion_matrix(dataset, model, normalize=False):\n",
    "    \"\"\"The confusion matrix is a table that is often used to describe the performance of a classification model on a set of data for which the true values are known.\n",
    "\n",
    "    The confusion matrix is a 2x2 table that contains 4 values:\n",
    "\n",
    "    - True Positive (TP): the number of correct positive predictions\n",
    "    - True Negative (TN): the number of correct negative predictions\n",
    "    - False Positive (FP): the number of incorrect positive predictions\n",
    "    - False Negative (FN): the number of incorrect negative predictions\n",
    "\n",
    "    The confusion matrix can be used to assess the holistic performance of a classification model by showing the accuracy, precision, recall, and F1 score of the model on a single figure.\n",
    "    \"\"\"\n",
    "    y_true = dataset.y\n",
    "    y_pred = dataset.y_pred(model=model)\n",
    "\n",
    "    if normalize:\n",
    "        confusion_matrix = metrics.confusion_matrix(y_true, y_pred, normalize=\"all\")\n",
    "    else:\n",
    "        confusion_matrix = metrics.confusion_matrix(y_true, y_pred)\n",
    "\n",
    "    cm_display = metrics.ConfusionMatrixDisplay(\n",
    "        confusion_matrix=confusion_matrix, display_labels=[False, True]\n",
    "    )\n",
    "    cm_display.plot()\n",
    "\n",
    "    plt.close()  # close the plot to avoid displaying it\n",
    "\n",
    "    return cm_display.figure_  # return the figure object itself"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Test dataset with normalize=True\n",
    "result = vm.tests.run_test(\n",
    "    \"my_custom_tests.ConfusionMatrix:test_dataset_normalized\",\n",
    "    inputs={\"model\": vm_model, \"dataset\": vm_test_ds},\n",
    "    params={\"normalize\": True},\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc2_4_2_'></a>\n",
    "\n",
    "#### Add a local test provider\n",
    "\n",
    "Finally, let's save our custom inline test to our local test provider:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create custom tests folder\n",
    "tests_folder = \"my_tests\"\n",
    "\n",
    "import os\n",
    "\n",
    "# create tests folder\n",
    "os.makedirs(tests_folder, exist_ok=True)\n",
    "\n",
    "# remove existing tests\n",
    "for f in os.listdir(tests_folder):\n",
    "    # remove files and pycache\n",
    "    if f.endswith(\".py\") or f == \"__pycache__\":\n",
    "        os.system(f\"rm -rf {tests_folder}/{f}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save custom inline test to custom tests folder\n",
    "confusion_matrix.save(\n",
    "    tests_folder,\n",
    "    imports=[\"import matplotlib.pyplot as plt\", \"from sklearn import metrics\"],\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Register local test provider\n",
    "from validmind.tests import LocalTestProvider\n",
    "\n",
    "# initialize the test provider with the tests folder we created earlier\n",
    "my_test_provider = LocalTestProvider(tests_folder)\n",
    "\n",
    "vm.tests.register_test_provider(\n",
    "    namespace=\"my_test_provider\",\n",
    "    test_provider=my_test_provider,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc3_'></a>\n",
    "\n",
    "## Reconnect to ValidMind\n",
    "\n",
    "After you insert test-driven blocks into your model documentation, changes should persist and become available every time you call [`vm.preview_template()`](https://docs.validmind.ai/validmind/validmind.html#preview_template).\n",
    "\n",
    "However, you'll need to reload the connection to the ValidMind Platform if you have added test-driven blocks when the connection was already established using [`reload()`](https://docs.validmind.ai/validmind/validmind.html#reload):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vm.reload()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, when you run `preview_template()` again, the three test-driven blocks you added to your documentation in the last two notebooks in should show up in the template in sections **2.3 Correlations and Interactions** and **3.2 Model Evaluation**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vm.preview_template()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc4_'></a>\n",
    "\n",
    "## Include custom test results\n",
    "\n",
    "Since your custom test IDs are now part of your documentation template, you can now run tests for an entire section and all additional custom tests should be loaded without any issues.\n",
    "\n",
    "Let's run all tests in the Model Evaluation section of the documentation. Note that we have been running the sample custom confusion matrix with `normalize=True` to demonstrate the ability to provide custom parameters.\n",
    "\n",
    "In the **Run the model evaluation tests** section of **[2 — Start the model development process](2-start_development_process.ipynb)**, you learned how to assign inputs to individual tests with [`run_documentation_tests()`](https://docs.validmind.ai/validmind/validmind.html#run_documentation_tests). Assigning parameters is similar, you only need to provide assign a `params` dictionary to a given test ID, `my_test_provider.ConfusionMatrix` in this case.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_config = {\n",
    "    \"validmind.model_validation.sklearn.ClassifierPerformance:in_sample\": {\n",
    "        \"inputs\": {\n",
    "            \"dataset\": vm_train_ds,\n",
    "            \"model\": vm_model,\n",
    "        },\n",
    "    },\n",
    "    \"my_test_provider.ConfusionMatrix\": {\n",
    "        \"params\": {\"normalize\": True},\n",
    "        \"inputs\": {\"dataset\": vm_test_ds, \"model\": vm_model},\n",
    "    },\n",
    "}\n",
    "results = vm.run_documentation_tests(\n",
    "    section=[\"model_evaluation\"],\n",
    "    inputs={\n",
    "        \"dataset\": vm_test_ds,  # Any test that requires a single dataset will use vm_test_ds\n",
    "        \"model\": vm_model,\n",
    "        \"datasets\": (\n",
    "            vm_train_ds,\n",
    "            vm_test_ds,\n",
    "        ),  # Any test that requires multiple datasets will use vm_train_ds and vm_test_ds\n",
    "    },\n",
    "    config=test_config,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc5_'></a>\n",
    "\n",
    "## Documentation template configuration\n",
    "\n",
    "Let's call the utility function [`vm.get_test_suite().get_default_config()`](https://docs.validmind.ai/validmind/validmind/vm_models.html#TestSuite.get_default_config) which will return the **default configuration for the entire documentation template as a dictionary:**\n",
    "\n",
    "- This configuration will contain all the test IDs and their default parameters.\n",
    "- You can then modify this configuration as needed and pass it to `run_documentation_tests()` to run all tests in the documentation template if needed.\n",
    "- You still have the option to continue running tests for one section at a time; `get_default_config()` simply provides a useful reference for providing default parameters to every test."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "\n",
    "model_test_suite = vm.get_test_suite()\n",
    "config = model_test_suite.get_default_config()\n",
    "print(\"Suite Config: \\n\", json.dumps(config, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "<a id='toc5_1_'></a>\n",
    "\n",
    "### Update the config\n",
    "\n",
    "The default config does not assign any inputs to a test, but you can assign inputs to individual tests as needed depending on the datasets and models you want to pass to individual tests.\n",
    "\n",
    "For this particular documentation template (binary classification), the ValidMind Library provides a sample configuration that can be used to populate the entire model documentation using the following inputs as placeholders:\n",
    "\n",
    "- A **`raw_dataset`** raw dataset\n",
    "- A **`train_dataset`** training dataset\n",
    "- A **`test_dataset`** test dataset\n",
    "- A trained **`model`** instance\n",
    "\n",
    "As part of updating the `config` you will need to ensure the correct `input_id`s are used in the final config passed to `run_documentation_tests()`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "metadata": {}
   },
   "outputs": [],
   "source": [
    "from validmind.datasets.classification import customer_churn\n",
    "from validmind.utils import preview_test_config\n",
    "\n",
    "test_config = customer_churn.get_demo_test_config()\n",
    "preview_test_config(test_config)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using this sample configuration, let's finish populating model documentation by running all tests for the Model Development section of the documentation.\n",
    "\n",
    "Recall that the training and test datasets in our exercise have the following `input_id` values:\n",
    "\n",
    "- **`train_dataset_final`** for the training dataset\n",
    "- **`test_dataset_final`** for the test dataset\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "config = {\n",
    "    \"validmind.model_validation.ModelMetadata\": {\n",
    "        \"inputs\": {\"model\": \"log_reg_model_v1\"},\n",
    "    },\n",
    "    \"validmind.data_validation.DatasetSplit\": {\n",
    "        \"inputs\": {\"datasets\": [\"train_dataset_final\", \"test_dataset_final\"]},\n",
    "    },\n",
    "    \"validmind.model_validation.sklearn.PopulationStabilityIndex\": {\n",
    "        \"inputs\": {\n",
    "            \"model\": \"log_reg_model_v1\",\n",
    "            \"datasets\": [\"train_dataset_final\", \"test_dataset_final\"],\n",
    "        },\n",
    "        \"params\": {\"num_bins\": 10, \"mode\": \"fixed\"},\n",
    "    },\n",
    "    \"validmind.model_validation.sklearn.ConfusionMatrix\": {\n",
    "        \"inputs\": {\"model\": \"log_reg_model_v1\", \"dataset\": \"test_dataset_final\"},\n",
    "    },\n",
    "    \"my_test_provider.ConfusionMatrix\": {\n",
    "        \"inputs\": {\"dataset\": \"test_dataset_final\", \"model\": \"log_reg_model_v1\"},\n",
    "    },\n",
    "    \"my_custom_tests.ConfusionMatrix:test_dataset_normalized\": {\n",
    "        \"inputs\": {\"dataset\": \"test_dataset_final\", \"model\": \"log_reg_model_v1\"},\n",
    "    },\n",
    "    \"validmind.model_validation.sklearn.ClassifierPerformance:in_sample\": {\n",
    "        \"inputs\": {\"model\": \"log_reg_model_v1\", \"dataset\": \"train_dataset_final\"}\n",
    "    },\n",
    "    \"validmind.model_validation.sklearn.ClassifierPerformance:out_of_sample\": {\n",
    "        \"inputs\": {\"model\": \"log_reg_model_v1\", \"dataset\": \"test_dataset_final\"}\n",
    "    },\n",
    "    \"validmind.model_validation.sklearn.PrecisionRecallCurve\": {\n",
    "        \"inputs\": {\"model\": \"log_reg_model_v1\", \"dataset\": \"test_dataset_final\"},\n",
    "    },\n",
    "    \"validmind.model_validation.sklearn.ROCCurve\": {\n",
    "        \"inputs\": {\"model\": \"log_reg_model_v1\", \"dataset\": \"test_dataset_final\"},\n",
    "    },\n",
    "    \"validmind.model_validation.sklearn.TrainingTestDegradation\": {\n",
    "        \"inputs\": {\n",
    "            \"model\": \"log_reg_model_v1\",\n",
    "            \"datasets\": [\"train_dataset_final\", \"test_dataset_final\"],\n",
    "        },\n",
    "        \"params\": {\n",
    "            \"metrics\": [\"accuracy\", \"precision\", \"recall\", \"f1\"],\n",
    "            \"max_threshold\": 0.1,\n",
    "        },\n",
    "    },\n",
    "    \"validmind.model_validation.sklearn.MinimumAccuracy\": {\n",
    "        \"inputs\": {\"model\": \"log_reg_model_v1\", \"dataset\": \"test_dataset_final\"},\n",
    "        \"params\": {\"min_threshold\": 0.7},\n",
    "    },\n",
    "    \"validmind.model_validation.sklearn.MinimumF1Score\": {\n",
    "        \"inputs\": {\"model\": \"log_reg_model_v1\", \"dataset\": \"test_dataset_final\"},\n",
    "        \"params\": {\"min_threshold\": 0.5},\n",
    "    },\n",
    "    \"validmind.model_validation.sklearn.MinimumROCAUCScore\": {\n",
    "        \"inputs\": {\"model\": \"log_reg_model_v1\", \"dataset\": \"test_dataset_final\"},\n",
    "        \"params\": {\"min_threshold\": 0.5},\n",
    "    },\n",
    "    \"validmind.model_validation.sklearn.PermutationFeatureImportance\": {\n",
    "        \"inputs\": {\"model\": \"log_reg_model_v1\", \"dataset\": \"test_dataset_final\"},\n",
    "    },\n",
    "    \"validmind.model_validation.sklearn.SHAPGlobalImportance\": {\n",
    "        \"inputs\": {\"model\": \"log_reg_model_v1\", \"dataset\": \"test_dataset_final\"},\n",
    "        \"params\": {\"kernel_explainer_samples\": 10},\n",
    "    },\n",
    "    \"validmind.model_validation.sklearn.WeakspotsDiagnosis\": {\n",
    "        \"inputs\": {\n",
    "            \"model\": \"log_reg_model_v1\",\n",
    "            \"datasets\": [\"train_dataset_final\", \"test_dataset_final\"],\n",
    "        },\n",
    "        \"params\": {\n",
    "            \"thresholds\": {\"accuracy\": 0.75, \"precision\": 0.5, \"recall\": 0.5, \"f1\": 0.7}\n",
    "        },\n",
    "    },\n",
    "    \"validmind.model_validation.sklearn.OverfitDiagnosis\": {\n",
    "        \"inputs\": {\n",
    "            \"model\": \"log_reg_model_v1\",\n",
    "            \"datasets\": [\"train_dataset_final\", \"test_dataset_final\"],\n",
    "        },\n",
    "        \"params\": {\"cut_off_percentage\": 4},\n",
    "    },\n",
    "    \"validmind.model_validation.sklearn.RobustnessDiagnosis\": {\n",
    "        \"inputs\": {\n",
    "            \"model\": \"log_reg_model_v1\",\n",
    "            \"datasets\": [\"train_dataset_final\", \"test_dataset_final\"],\n",
    "        },\n",
    "        \"params\": {\n",
    "            \"scaling_factor_std_dev_list\": [0.0, 0.1, 0.2, 0.3, 0.4, 0.5],\n",
    "            \"accuracy_decay_threshold\": 4,\n",
    "        },\n",
    "    },\n",
    "}\n",
    "\n",
    "\n",
    "full_suite = vm.run_documentation_tests(\n",
    "    section=\"model_development\",\n",
    "    config=config,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc6_'></a>\n",
    "\n",
    "## In summary\n",
    "\n",
    "In this final notebook, you learned how to:\n",
    "\n",
    "- [x] Refresh the connection from the ValidMind Library to the ValidMind Platform after you've inserted test-driven blocks to your documentation\n",
    "- [x] Include custom test results in your model documentation\n",
    "- [x] View and configure the configuration for your model documentation template\n",
    "\n",
    "With our ValidMind for model development series of notebooks, you learned how to document a model end-to-end with the ValidMind Library by running through some common scenarios in a typical model development setting:\n",
    "\n",
    "- Running out-of-the-box tests\n",
    "- Documenting your model by adding evidence to model documentation\n",
    "- Extending the capabilities of the ValidMind Library by implementing custom tests\n",
    "- Ensuring that the documentation is complete by running all tests in the documentation template"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc7_'></a>\n",
    "\n",
    "## Next steps"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc7_1_'></a>\n",
    "\n",
    "### Work with your model documentation\n",
    "\n",
    "Now that you've logged all your test results and generated a draft for your model documentation, head to the ValidMind Platform to wrap up your model documentation. Continue to work on your model documentation by:\n",
    "\n",
    "- **Run and log more tests:** Use the skills you learned in this series of notebooks to run and log more individual tests, including custom tests, then insert them into your documentation as supplementary evidence. (Learn more: [`validmind.tests`](https://docs.validmind.ai/validmind/validmind/tests.html))\n",
    "\n",
    "- **Inserting additional test results:** Add **Test-Driven Blocks** under any relevant section of your model documentation. (Learn more: [Work with test results](https://docs.validmind.ai/guide/model-documentation/work-with-test-results.html))\n",
    "\n",
    "- **Making qualitative edits to your test descriptions:** Click on the description of any inserted test results to review and edit the ValidMind-generated test descriptions for quality and accuracy. (Learn more: [Working with model documentation](https://docs.validmind.ai/guide/model-documentation/working-with-model-documentation.html#add-or-edit-documentation))\n",
    "\n",
    "- **View guidelines:** In any section of your model documentation, click **​ValidMind Insights** in the top right corner to reveal the Documentation Guidelines for each section to help guide the contents of your model documentation. (Learn more: [View documentation guidelines](https://docs.validmind.ai/guide/model-documentation/view-documentation-guidelines.html))\n",
    "\n",
    "- **Collaborate with other stakeholders:** Use the ValidMind Platform's real-time collaborative features to work seamlessly together with the rest of your organization, including model validators. Review suggested changes in your content blocks, work with versioned history, and use comments to discuss specific portions of your model documentation. (Learn more: [Collaborate with others](https://docs.validmind.ai/guide/model-documentation/collaborate-with-others.html))\n",
    "\n",
    "When your model documentation is complete and ready for review, submit it for approval from the same ValidMind Platform where you made your edits and collaborated with the rest of your organization, ensuring transparency and a thorough model development history. (Learn more: [Submit for approval](https://docs.validmind.ai/guide/model-documentation/submit-for-approval.html))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc7_2_'></a>\n",
    "\n",
    "### Learn more\n",
    "\n",
    "Now that you're familiar with the basics, you can explore the following notebooks to get a deeper understanding on how the ValidMind Library allows you generate model documentation for any use case:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc7_2_1_'></a>\n",
    "\n",
    "#### Use cases\n",
    "\n",
    "- [Application scorecard demo](../../code_samples/credit_risk/application_scorecard_demo.ipynb)\n",
    "- [Linear regression documentation demo](../../code_samples/regression/quickstart_regression_full_suite.ipynb)\n",
    "- [LLM model documentation demo](../../code_samples/nlp_and_llm/foundation_models_integration_demo.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc7_2_2_'></a>\n",
    "\n",
    "#### More how-to guides and code samples\n",
    "\n",
    "- [Explore available tests in detail](../../how_to/explore_tests.ipynb)\n",
    "- [In-depth guide on running dataset based tests](../../how_to/run_tests/1_run_dataset_based_tests.ipynb)\n",
    "- [In-depth guide for implementing custom tests](../../code_samples/custom_tests/implement_custom_tests.ipynb)\n",
    "- [In-depth guide to external test providers](../../code_samples/custom_tests/integrate_external_test_providers.ipynb)\n",
    "- [Configuring dataset features](../../how_to/configure_dataset_features.ipynb)\n",
    "- [Introduction to unit and composite metrics](../../how_to/run_unit_metrics.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc7_2_3_'></a>\n",
    "\n",
    "#### Discover more learning resources\n",
    "\n",
    "All notebook samples can be found in the following directories of the ValidMind Library GitHub repository:\n",
    "\n",
    "- [Code samples](https://github.com/validmind/validmind-library/tree/main/notebooks/code_samples)\n",
    "- [How-to guides](https://github.com/validmind/validmind-library/tree/main/notebooks/how_to)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "ValidMind Library",
   "language": "python",
   "name": "validmind"
  },
  "language_info": {
   "name": "python",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
