{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# ValidMind for model development 2 — Start the model development process\n",
    "\n",
    "Learn how to use ValidMind for your end-to-end model documentation process with our series of four introductory notebooks. In this second notebook, you'll run tests and investigate results, then add the results or evidence to your documentation.\n",
    "\n",
    "You'll become familiar with the individual tests available in ValidMind, as well as how to run them and change parameters as necessary. Using ValidMind's repository of individual tests as building blocks helps you ensure that a model is being built appropriately. \n",
    "\n",
    "**For a full list of out-of-the-box tests,** refer to our [Test descriptions](https://docs.validmind.ai/developer/model-testing/test-descriptions.html) or try the interactive [Test sandbox](https://docs.validmind.ai/developer/model-testing/test-sandbox.html).\n",
    "\n",
    "<div class=\"alert alert-block alert-info\" style=\"background-color: #B5B5B510; color: black; border: 1px solid #083E44; border-left-width: 5px; box-shadow: 2px 2px 4px rgba(0, 0, 0, 0.2);border-radius: 5px;\"><span style=\"color: #083E44;\"><b>Learn by doing</b></span>\n",
    "<br></br>\n",
    "Our course tailor-made for developers new to ValidMind combines this series of notebooks with more a more in-depth introduction to the ValidMind Platform — <a href=\"https://docs.validmind.ai/training/developer-fundamentals/developer-fundamentals-register.html\" style=\"color: #DE257E;\"><b>Developer Fundamentals</b></a></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "::: {.content-hidden when-format=\"html\"}\n",
    "## Contents    \n",
    "- [Prerequisites](#toc1_)    \n",
    "- [Setting up](#toc2_)    \n",
    "  - [Initialize the ValidMind Library](#toc2_1_)    \n",
    "  - [Import sample dataset](#toc2_2_)    \n",
    "  - [Identify qualitative tests](#toc2_3_)    \n",
    "  - [Initialize the ValidMind datasets](#toc2_4_)    \n",
    "- [Running tests](#toc3_)    \n",
    "  - [Run tabular data tests](#toc3_1_)    \n",
    "  - [Utilize test output](#toc3_2_)    \n",
    "- [Documenting test results](#toc4_)    \n",
    "  - [Run and log multiple tests](#toc4_1_)    \n",
    "  - [Run and log an individual test](#toc4_2_)    \n",
    "    - [Add individual test results to model documentation](#toc4_2_1_)    \n",
    "- [Model testing](#toc5_)    \n",
    "  - [Train simple logistic regression model](#toc5_1_)    \n",
    "  - [Initialize model evaluation objects](#toc5_2_)    \n",
    "  - [Assign predictions](#toc5_3_)    \n",
    "  - [Run the model evaluation tests](#toc5_4_)    \n",
    "- [In summary](#toc6_)    \n",
    "- [Next steps](#toc7_)    \n",
    "  - [Integrate custom tests](#toc7_1_)    \n",
    "\n",
    ":::\n",
    "<!-- jn-toc-notebook-config\n",
    "\tnumbering=false\n",
    "\tanchor=true\n",
    "\tflat=false\n",
    "\tminLevel=2\n",
    "\tmaxLevel=4\n",
    "\t/jn-toc-notebook-config -->\n",
    "<!-- THIS CELL WILL BE REPLACED ON TOC UPDATE. DO NOT WRITE YOUR TEXT IN THIS CELL -->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc1_'></a>\n",
    "\n",
    "## Prerequisites\n",
    "\n",
    "In order to log test results or evidence to your model documentation with this notebook, you'll need to first have:\n",
    "\n",
    "- [x] Registered a model within the ValidMind Platform with a predefined documentation template\n",
    "- [x] Installed the ValidMind Library in your local environment, allowing you to access all its features\n",
    "\n",
    "<div class=\"alert alert-block alert-info\" style=\"background-color: #B5B5B510; color: black; border: 1px solid #083E44; border-left-width: 5px; box-shadow: 2px 2px 4px rgba(0, 0, 0, 0.2);border-radius: 5px;\"><span style=\"color: #083E44;\"><b>Need help with the above steps?</b></span>\n",
    "<br></br>\n",
    "Refer to the first notebook in this series: <a href=\"1-set_up_validmind.ipynb\" style=\"color: #DE257E;\"><b>1 — Set up the ValidMind Library</b></a></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc2_'></a>\n",
    "\n",
    "## Setting up"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc2_1_'></a>\n",
    "\n",
    "### Initialize the ValidMind Library\n",
    "\n",
    "First, let's connect up the ValidMind Library to our model we previously registered in the ValidMind Platform:\n",
    "\n",
    "1. In a browser, [log in to ValidMind](https://docs.validmind.ai/guide/configuration/log-in-to-validmind.html).\n",
    "\n",
    "2. In the left sidebar, navigate to **Inventory** and select the model you registered for this \"ValidMind for model development\" series of notebooks.\n",
    "\n",
    "3. Go to **Getting Started** and click **Copy snippet to clipboard**.\n",
    "\n",
    "Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Make sure the ValidMind Library is installed\n",
    "\n",
    "%pip install -q validmind\n",
    "\n",
    "# Load your model identifier credentials from an `.env` file\n",
    "\n",
    "%load_ext dotenv\n",
    "%dotenv .env\n",
    "\n",
    "# Or replace with your code snippet\n",
    "\n",
    "import validmind as vm\n",
    "\n",
    "vm.init(\n",
    "    # api_host=\"...\",\n",
    "    # api_key=\"...\",\n",
    "    # api_secret=\"...\",\n",
    "    # model=\"...\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc2_2_'></a>\n",
    "\n",
    "### Import sample dataset\n",
    "\n",
    "Then, let's import the public [Bank Customer Churn Prediction](https://www.kaggle.com/datasets/shantanudhakadd/bank-customer-churn-prediction) dataset from Kaggle. \n",
    "\n",
    "In our below example, note that: \n",
    "\n",
    "- The target column, `Exited` has a value of `1` when a customer has churned and `0` otherwise.\n",
    "- The ValidMind Library provides a wrapper to automatically load the dataset as a [Pandas DataFrame](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.html) object. A Pandas Dataframe is a two-dimensional tabular data structure that makes use of rows and columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from validmind.datasets.classification import customer_churn as demo_dataset\n",
    "\n",
    "print(\n",
    "    f\"Loaded demo dataset with: \\n\\n\\t• Target column: '{demo_dataset.target_column}' \\n\\t• Class labels: {demo_dataset.class_labels}\"\n",
    ")\n",
    "\n",
    "raw_df = demo_dataset.load_data()\n",
    "raw_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc2_3_'></a>\n",
    "\n",
    "### Identify qualitative tests\n",
    "\n",
    "Next, let's say we want to do some data quality assessments by running a few individual tests.\n",
    "\n",
    "Use the [`vm.tests.list_tests()` function](https://docs.validmind.ai/validmind/validmind/tests.html#list_tests) introduced by the first notebook in this series in combination with [`vm.tests.list_tags()`](https://docs.validmind.ai/validmind/validmind/tests.html#list_tags) and [`vm.tests.list_tasks()`](https://docs.validmind.ai/validmind/validmind/tests.html#list_tasks) to find which prebuilt tests are relevant for data quality assessment:\n",
    "\n",
    "- **`tasks`** represent the kind of modeling task associated with a test. Here we'll focus on `classification` tasks.\n",
    "- **`tags`** are free-form descriptions providing more details about the test, for example, what category the test falls into. Here we'll focus on the `data_quality` tag.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the list of available task types\n",
    "sorted(vm.tests.list_tasks())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the list of available tags\n",
    "sorted(vm.tests.list_tags())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can pass `tags` and `tasks` as parameters to the `vm.tests.list_tests()` function to filter the tests based on the tags and task types.\n",
    "\n",
    "For example, to find tests related to tabular data quality for classification models, you can call `list_tests()` like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vm.tests.list_tests(task=\"classification\", tags=[\"tabular_data\", \"data_quality\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\" style=\"background-color: #B5B5B510; color: black; border: 1px solid #083E44; border-left-width: 5px; box-shadow: 2px 2px 4px rgba(0, 0, 0, 0.2);border-radius: 5px;\"><span style=\"color: #083E44;\"><b>Want to learn more about navigating ValidMind tests?</b></span>\n",
    "<br></br>\n",
    "Refer to our notebook outlining the utilities available for viewing and understanding available ValidMind tests: <a href=\"https://docs.validmind.ai/notebooks/how_to/explore_tests.html\" style=\"color: #DE257E;\"><b>Explore tests</b></a></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc2_4_'></a>\n",
    "\n",
    "### Initialize the ValidMind datasets\n",
    "\n",
    "With the individual tests we want to run identified, the next step is to connect your data with a ValidMind `Dataset` object. **This step is always necessary every time you want to connect a dataset to documentation and produce test results through ValidMind,** but you only need to do it once per dataset.\n",
    "\n",
    "Initialize a ValidMind dataset object using the [`init_dataset` function](https://docs.validmind.ai/validmind/validmind.html#init_dataset) from the ValidMind (`vm`) module. For this example, we'll pass in the following arguments:\n",
    "\n",
    "- **`dataset`** — The raw dataset that you want to provide as input to tests.\n",
    "- **`input_id`** — A unique identifier that allows tracking what inputs are used when running each individual test.\n",
    "- **`target_column`** — A required argument if tests require access to true values. This is the name of the target column in the dataset.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# vm_raw_dataset is now a VMDataset object that you can pass to any ValidMind test\n",
    "vm_raw_dataset = vm.init_dataset(\n",
    "    dataset=raw_df,\n",
    "    input_id=\"raw_dataset\",\n",
    "    target_column=\"Exited\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc3_'></a>\n",
    "\n",
    "## Running tests\n",
    "\n",
    "Now that we know how to initialize a ValidMind `dataset` object, we're ready to run some tests!\n",
    "\n",
    "You run individual tests by calling [the `run_test` function](https://docs.validmind.ai/validmind/validmind/tests.html#run_test) provided by the `validmind.tests` module. For the examples below, we'll pass in the following arguments:\n",
    "\n",
    "- **`test_id`** — The ID of the test to run, as seen in the `ID` column when you run `list_tests`. \n",
    "- **`params`** — A dictionary of parameters for the test. These will override any `default_params` set in the test definition. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc3_1_'></a>\n",
    "\n",
    "### Run tabular data tests\n",
    "\n",
    "The inputs expected by a test can also be found in the test definition — let's take [`validmind.data_validation.DescriptiveStatistics`](https://docs.validmind.ai/tests/data_validation/DescriptiveStatistics.html) as an example.\n",
    "\n",
    "Note that the output of the [`describe_test()` function](https://docs.validmind.ai/validmind/validmind/tests.html#describe_test) below shows that this test expects a `dataset` as input:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vm.tests.describe_test(\"validmind.data_validation.DescriptiveStatistics\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's run a few tests to assess the quality of the dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "result = vm.tests.run_test(\n",
    "    test_id=\"validmind.data_validation.DescriptiveStatistics\",\n",
    "    inputs={\"dataset\": vm_raw_dataset},\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "result2 = vm.tests.run_test(\n",
    "    test_id=\"validmind.data_validation.ClassImbalance\",\n",
    "    inputs={\"dataset\": vm_raw_dataset},\n",
    "    params={\"min_percent_threshold\": 30},\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The output above shows that [the class imbalance test](https://docs.validmind.ai/tests/data_validation/ClassImbalance.html) did not pass according to the value we set for `min_percent_threshold`.\n",
    "\n",
    "To address this issue, we'll re-run the test on some processed data. In this case let's apply a very simple rebalancing technique to the dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "raw_copy_df = raw_df.sample(frac=1)  # Create a copy of the raw dataset\n",
    "\n",
    "# Create a balanced dataset with the same number of exited and not exited customers\n",
    "exited_df = raw_copy_df.loc[raw_copy_df[\"Exited\"] == 1]\n",
    "not_exited_df = raw_copy_df.loc[raw_copy_df[\"Exited\"] == 0].sample(n=exited_df.shape[0])\n",
    "\n",
    "balanced_raw_df = pd.concat([exited_df, not_exited_df])\n",
    "balanced_raw_df = balanced_raw_df.sample(frac=1, random_state=42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With this new balanced dataset, you can re-run the individual test to see if it now passes the class imbalance test requirement.\n",
    "\n",
    "As this is technically a different dataset, **remember to first initialize a new ValidMind `Dataset` object** to pass in as input as required by `run_test()`:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Register new data and now 'balanced_raw_dataset' is the new dataset object of interest\n",
    "vm_balanced_raw_dataset = vm.init_dataset(\n",
    "    dataset=balanced_raw_df,\n",
    "    input_id=\"balanced_raw_dataset\",\n",
    "    target_column=\"Exited\",\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Pass the initialized `balanced_raw_dataset` as input into the test run\n",
    "result = vm.tests.run_test(\n",
    "    test_id=\"validmind.data_validation.ClassImbalance\",\n",
    "    inputs={\"dataset\": vm_balanced_raw_dataset},\n",
    "    params={\"min_percent_threshold\": 30},\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "<a id='toc3_2_'></a>\n",
    "\n",
    "### Utilize test output\n",
    "\n",
    "You can utilize the output from a ValidMind test for further use, for example, if you want to remove highly correlated features. Removing highly correlated features helps make the model simpler, more stable, and easier to understand.\n",
    "\n",
    "Below we demonstrate how to retrieve the list of features with the highest correlation coefficients and use them to reduce the final list of features for modeling.\n",
    "\n",
    "First, we'll run [`validmind.data_validation.HighPearsonCorrelation`](https://docs.validmind.ai/tests/data_validation/HighPearsonCorrelation.html) with the `balanced_raw_dataset` we initialized previously as input as is for comparison with later runs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "metadata": {}
   },
   "outputs": [],
   "source": [
    "corr_result = vm.tests.run_test(\n",
    "    test_id=\"validmind.data_validation.HighPearsonCorrelation\",\n",
    "    params={\"max_threshold\": 0.3},\n",
    "    inputs={\"dataset\": vm_balanced_raw_dataset},\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The output above shows that the test did not pass according to the value we set for `max_threshold`.\n",
    "\n",
    "`corr_result` is an object of type `TestResult`. We can inspect the result object to see what the test has produced:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "metadata": {}
   },
   "outputs": [],
   "source": [
    "print(type(corr_result))\n",
    "print(\"Result ID: \", corr_result.result_id)\n",
    "print(\"Params: \", corr_result.params)\n",
    "print(\"Passed: \", corr_result.passed)\n",
    "print(\"Tables: \", corr_result.tables)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's remove the highly correlated features and create a new VM `dataset` object.\n",
    "\n",
    "We'll begin by checking out the table in the result and extracting a list of features that failed the test:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "metadata": {}
   },
   "outputs": [],
   "source": [
    "# Extract table from `corr_result.tables`\n",
    "features_df = corr_result.tables[0].data\n",
    "features_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "metadata": {}
   },
   "outputs": [],
   "source": [
    "# Extract list of features that failed the test\n",
    "high_correlation_features = features_df[features_df[\"Pass/Fail\"] == \"Fail\"][\"Columns\"].tolist()\n",
    "high_correlation_features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, extract the feature names from the list of strings (example: `(Age, Exited)` > `Age`):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "high_correlation_features = [feature.split(\",\")[0].strip(\"()\") for feature in high_correlation_features]\n",
    "high_correlation_features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, it's time to re-initialize the dataset with the highly correlated features removed.\n",
    "\n",
    "**Note the use of a different `input_id`.** This allows tracking the inputs used when running each individual test."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "metadata": {}
   },
   "outputs": [],
   "source": [
    "# Remove the highly correlated features from the dataset\n",
    "balanced_raw_no_age_df = balanced_raw_df.drop(columns=high_correlation_features)\n",
    "\n",
    "# Re-initialize the dataset object\n",
    "vm_raw_dataset_preprocessed = vm.init_dataset(\n",
    "    dataset=balanced_raw_no_age_df,\n",
    "    input_id=\"raw_dataset_preprocessed\",\n",
    "    target_column=\"Exited\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Re-running the test with the reduced feature set should pass the test:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "metadata": {}
   },
   "outputs": [],
   "source": [
    "corr_result = vm.tests.run_test(\n",
    "    test_id=\"validmind.data_validation.HighPearsonCorrelation\",\n",
    "    params={\"max_threshold\": 0.3},\n",
    "    inputs={\"dataset\": vm_raw_dataset_preprocessed},\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also plot the correlation matrix to visualize the new correlation between features:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "corr_result = vm.tests.run_test(\n",
    "    test_id=\"validmind.data_validation.PearsonCorrelationMatrix\",\n",
    "    inputs={\"dataset\": vm_raw_dataset_preprocessed},\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc4_'></a>\n",
    "\n",
    "## Documenting test results\n",
    "\n",
    "Now that we've done some analysis on two different datasets, we can use ValidMind to easily document why certain things were done to our raw data with testing to support it.\n",
    "\n",
    "Every test result returned by the `run_test()` function has a [`.log()` method](https://docs.validmind.ai/validmind/validmind/vm_models.html#TestResult.log) that can be used to send the test results to the ValidMind Platform:\n",
    "\n",
    "- When using `run_documentation_tests()`, documentation sections will be automatically populated with the results of all tests registered in the documentation template.\n",
    "- When logging individual test results to the platform, you'll need to manually add those results to the desired section of the model documentation.\n",
    "\n",
    "To demonstrate how to add test results to your model documentation, we'll populate the entire **Data Preparation** section of the documentation using the clean `vm_raw_dataset_preprocessed` dataset as input, and then document an additional individual result for the highly correlated dataset `vm_balanced_raw_dataset`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "<a id='toc4_1_'></a>\n",
    "\n",
    "### Run and log multiple tests\n",
    "\n",
    "[`run_documentation_tests()`](https://docs.validmind.ai/validmind/validmind.html#run_documentation_tests) allows you to run multiple tests at once and automatically log the results to your documentation. Below, we'll run the tests using the previously initialized `vm_raw_dataset_preprocessed` as input — this will populate the entire **Data Preparation** section for every test that is part of the documentation template.\n",
    "\n",
    "For this example, we'll pass in the following arguments:\n",
    "\n",
    "- **`inputs`:** Any inputs to be passed to the tests.\n",
    "- **`config`:** A dictionary `<test_id>:<test_config>` that allows configuring each test individually. Each test config requires the following:\n",
    "  - **`params`:** Individual test parameters.\n",
    "  - **`inputs`:** Individual test inputs. This overrides any inputs passed from the `run_documentation_tests()` function.\n",
    "\n",
    "When including explicit configuration for individual tests, you'll need to specify the `inputs` even if they mirror what is included in your global configuration.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Individual test config with inputs specified\n",
    "test_config = {\n",
    "    \"validmind.data_validation.ClassImbalance\": {\n",
    "        \"params\": {\"min_percent_threshold\": 30},\n",
    "        \"inputs\": {\"dataset\": vm_raw_dataset_preprocessed},\n",
    "    },\n",
    "    \"validmind.data_validation.HighPearsonCorrelation\": {\n",
    "        \"params\": {\"max_threshold\": 0.3},\n",
    "        \"inputs\": {\"dataset\": vm_raw_dataset_preprocessed},\n",
    "    },\n",
    "}\n",
    "\n",
    "# Global test config\n",
    "tests_suite = vm.run_documentation_tests(\n",
    "    inputs={\n",
    "        \"dataset\": vm_raw_dataset_preprocessed,\n",
    "    },\n",
    "    config=test_config,\n",
    "    section=[\"data_preparation\"],\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc4_2_'></a>\n",
    "\n",
    "### Run and log an individual test\n",
    "\n",
    "Next, we'll use the previously initialized `vm_balanced_raw_dataset` (that still has a highly correlated `Age` column) as input to run an individual test, then log the result to the ValidMind Platform.\n",
    "\n",
    "When running individual tests, **you can use a custom `result_id` to tag the individual result with a unique identifier:** \n",
    "\n",
    "- This `result_id` can be appended to `test_id` with a `:` separator.\n",
    "- The `balanced_raw_dataset` result identifier will correspond to the `balanced_raw_dataset` input, the dataset that still has the `Age` column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "result = vm.tests.run_test(\n",
    "    test_id=\"validmind.data_validation.HighPearsonCorrelation:balanced_raw_dataset\",\n",
    "    params={\"max_threshold\": 0.3},\n",
    "    inputs={\"dataset\": vm_balanced_raw_dataset},\n",
    ")\n",
    "result.log()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\" style=\"background-color: #B5B5B510; color: black; border: 1px solid #083E44; border-left-width: 5px; box-shadow: 2px 2px 4px rgba(0, 0, 0, 0.2);border-radius: 5px;\"><span style=\"color: #083E44;\"><b>Note the output returned indicating that a test-driven block doesn't currently exist in your model's documentation for this particular test ID. </b></span>\n",
    "<br></br>\n",
    "That's expected, as when we run individual tests the results logged need to be manually added to your documentation within the ValidMind Platform.</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc4_2_1_'></a>\n",
    "\n",
    "#### Add individual test results to model documentation\n",
    "\n",
    "With the test results logged, let's head to the model we connected to at the beginning of this notebook and insert our test results into the documentation ([Need more help?](https://docs.validmind.ai/developer/model-documentation/work-with-test-results.html)):\n",
    "\n",
    "1. From the **Inventory** in the ValidMind Platform, go to the model you connected to earlier.\n",
    "\n",
    "2. In the left sidebar that appears for your model, click **Documentation**.\n",
    "\n",
    "3. Locate the Data Preparation section and click on **2.3. Correlations and Interactions** to expand that section.\n",
    "\n",
    "4. Hover under the Pearson Correlation Matrix content block until a horizontal dashed line with a **+** button appears, indicating that you can insert a new block.\n",
    "\n",
    "    <img src= \"add-content-block.gif\" alt=\"Screenshot showing insert block button in model documentation\" style=\"border: 2px solid #083E44; border-radius: 8px; border-right-width: 2px; border-bottom-width: 3px;\">\n",
    "    <br><br>\n",
    "\n",
    "5. Click **+** and then select **Test-Driven Block** under FROM LIBRARY:\n",
    "\n",
    "    - Click on **VM Library** under TEST-DRIVEN in the left sidebar.\n",
    "    - In the search bar, type in `HighPearsonCorrelation`.\n",
    "    - Select `HighPearsonCorrelation:balanced_raw_dataset` as the test.\n",
    "\n",
    "    A preview of the test gets shown:\n",
    "\n",
    "    <img src= \"selecting-high-pearson-correlation-test.png\" alt=\"Screenshot showing the HighPearsonCorrelation test selected\" style=\"border: 2px solid #083E44; border-radius: 8px; border-right-width: 2px; border-bottom-width: 3px;\">\n",
    "    <br><br>\n",
    "\n",
    "6. Finally, click **Insert 1 Test Result to Document** to add the test result to the documentation.\n",
    "\n",
    "    Confirm that the individual results for the high correlation test has been correctly inserted into section **2.3. Correlations and Interactions** of the documentation.\n",
    "\n",
    "7. Finalize the documentation by editing the test result's description block to explain the changes you made to the raw data and the reasons behind them as shown in the screenshot below:\n",
    "\n",
    "    <img src= \"high-pearson-correlation-block.png\" alt=\"Screenshot showing the inserted High Pearson Correlation block\" style=\"border: 2px solid #083E44; border-radius: 8px; border-right-width: 2px; border-bottom-width: 3px;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc5_'></a>\n",
    "\n",
    "## Model testing\n",
    "\n",
    "So far, we've focused on the data assessment and pre-processing that usually occurs prior to any models being built. Now, let's instead assume we have already built a model and we want to incorporate some model results into our documentation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc5_1_'></a>\n",
    "\n",
    "### Train simple logistic regression model\n",
    "\n",
    "Using ValidMind tests, we'll train a simple logistic regression model on our dataset and evaluate its performance by using the `LogisticRegression` class from the `sklearn.linear_model`.\n",
    "\n",
    "To start, let's grab the first few rows from the `balanced_raw_no_age_df` dataset with the highly correlated features removed we initialized earlier:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "metadata": {}
   },
   "outputs": [],
   "source": [
    "balanced_raw_no_age_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before training the model, we need to encode the categorical features in the dataset:\n",
    "\n",
    "- Use the `OneHotEncoder` class from the `sklearn.preprocessing` module to encode the categorical features.\n",
    "- The categorical features in the dataset are `Geography` and `Gender`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "metadata": {}
   },
   "outputs": [],
   "source": [
    "balanced_raw_no_age_df = pd.get_dummies(\n",
    "    balanced_raw_no_age_df, columns=[\"Geography\", \"Gender\"], drop_first=True\n",
    ")\n",
    "balanced_raw_no_age_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll split our preprocessed dataset into training and testing, to help assess how well the model generalizes to unseen data:\n",
    "\n",
    "- We start by dividing our `balanced_raw_no_age_df` dataset into training and test subsets using `train_test_split`, with 80% of the data allocated to training (`train_df`) and 20% to testing (`test_df`).\n",
    "- From each subset, we separate the features (all columns except \"Exited\") into `X_train` and `X_test`, and the target column (\"Exited\") into `y_train` and `y_test`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "train_df, test_df = train_test_split(balanced_raw_no_age_df, test_size=0.20)\n",
    "\n",
    "X_train = train_df.drop(\"Exited\", axis=1)\n",
    "y_train = train_df[\"Exited\"]\n",
    "X_test = test_df.drop(\"Exited\", axis=1)\n",
    "y_test = test_df[\"Exited\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then using `GridSearchCV`, we'll find the best-performing hyperparameters or settings and save them:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "# Logistic Regression grid params\n",
    "log_reg_params = {\n",
    "    \"penalty\": [\"l1\", \"l2\"],\n",
    "    \"C\": [0.001, 0.01, 0.1, 1, 10, 100, 1000],\n",
    "    \"solver\": [\"liblinear\"],\n",
    "}\n",
    "\n",
    "# Grid search for Logistic Regression\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "grid_log_reg = GridSearchCV(LogisticRegression(), log_reg_params)\n",
    "grid_log_reg.fit(X_train, y_train)\n",
    "\n",
    "# Logistic Regression best estimator\n",
    "log_reg = grid_log_reg.best_estimator_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc5_2_'></a>\n",
    "\n",
    "### Initialize model evaluation objects\n",
    "\n",
    "The last step for evaluating the model's performance is to initialize the ValidMind `Dataset` and `Model` objects in preparation for assigning model predictions to each dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize the datasets into their own dataset objects\n",
    "vm_train_ds = vm.init_dataset(\n",
    "    input_id=\"train_dataset_final\",\n",
    "    dataset=train_df,\n",
    "    target_column=\"Exited\",\n",
    ")\n",
    "\n",
    "vm_test_ds = vm.init_dataset(\n",
    "    input_id=\"test_dataset_final\",\n",
    "    dataset=test_df,\n",
    "    target_column=\"Exited\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You'll also need to initialize a ValidMind model object (`vm_model`) that can be passed to other functions for analysis and tests on the data for each of our three models.\n",
    "\n",
    "You simply initialize this model object with [`vm.init_model()`](https://docs.validmind.ai/validmind/validmind.html#init_model):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Register the model\n",
    "vm_model = vm.init_model(log_reg, input_id=\"log_reg_model_v1\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc5_3_'></a>\n",
    "\n",
    "### Assign predictions\n",
    "\n",
    "Once the model has been registered you can assign model predictions to the training and testing datasets.\n",
    "\n",
    "- The [`assign_predictions()` method](https://docs.validmind.ai/validmind/validmind/vm_models.html#assign_predictions) from the `Dataset` object can link existing predictions to any number of models.\n",
    "- This method links the model's class prediction values and probabilities to our `vm_train_ds` and `vm_test_ds` datasets.\n",
    "\n",
    "If no prediction values are passed, the method will compute predictions automatically:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vm_train_ds.assign_predictions(model=vm_model)\n",
    "vm_test_ds.assign_predictions(model=vm_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc5_4_'></a>\n",
    "\n",
    "### Run the model evaluation tests\n",
    "\n",
    "In this next example, we'll focus on running the tests within the Model Development section of the model documentation. Only tests associated with this section will be executed, and the corresponding results will be updated in the model documentation.\n",
    "\n",
    "- Note the additional config that is passed to `run_documentation_tests()` — this allows you to override `inputs` or `params` in certain tests.\n",
    "- In our case, we want to explicitly use the `vm_train_ds` for the [`validmind.model_validation.sklearn.ClassifierPerformance:in_sample` test](https://docs.validmind.ai/tests/model_validation/sklearn/ClassifierPerformance.html), since it's supposed to run on the training dataset and not the test dataset.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_config = {\n",
    "    \"validmind.model_validation.sklearn.ClassifierPerformance:in_sample\": {\n",
    "        \"inputs\": {\n",
    "            \"dataset\": vm_train_ds,\n",
    "            \"model\": vm_model,\n",
    "        },\n",
    "    }\n",
    "}\n",
    "results = vm.run_documentation_tests(\n",
    "    section=[\"model_development\"],\n",
    "    inputs={\n",
    "        \"dataset\": vm_test_ds,  # Any test that requires a single dataset will use vm_test_ds\n",
    "        \"model\": vm_model,\n",
    "        \"datasets\": (\n",
    "            vm_train_ds,\n",
    "            vm_test_ds,\n",
    "        ),  # Any test that requires multiple datasets will use vm_train_ds and vm_test_ds\n",
    "    },\n",
    "    config=test_config,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc6_'></a>\n",
    "\n",
    "## In summary\n",
    "\n",
    "In this second notebook, you learned how to:\n",
    "\n",
    "- [x] Import a sample dataset\n",
    "- [x] Identify which tests you might want to run with ValidMind\n",
    "- [x] Initialize ValidMind datasets and model objects\n",
    "- [x] Run individual tests\n",
    "- [x] Utilize the output from tests you've run\n",
    "- [x] Log test results from sets of or individual tests as evidence to the ValidMind Platform\n",
    "- [x] Add supplementary individual test results to your documentation\n",
    "- [x] Assign model predictions to your ValidMind model objects\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc7_'></a>\n",
    "\n",
    "## Next steps"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc7_1_'></a>\n",
    "\n",
    "### Integrate custom tests\n",
    "\n",
    "Now that you're familiar with the basics of using the ValidMind Library to run and log tests to provide evidence for your model documentation, let's learn how to incorporate your own custom tests into ValidMind: **[3 — Integrate custom tests](3-integrate_custom_tests.ipynb)**"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "ValidMind Library",
   "language": "python",
   "name": "validmind"
  },
  "language_info": {
   "name": "python",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
