{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# ValidMind for model validation 2 — Start the model validation process\n",
    "\n",
    "Learn how to use ValidMind for your end-to-end model validation process with our series of four introductory notebooks. In this second notebook, independently verify the data quality tests performed on the dataset used to train the champion model.\n",
    "\n",
    "You'll learn how to run relevant validation tests with ValidMind, log the results of those tests to the ValidMind Platform, and insert your logged test results as evidence into your validation report. You'll become familiar with the tests available in ValidMind, as well as how to run them. Running tests during model validation is crucial to the effective challenge process, as we want to independently evaluate the evidence and assessments provided by the model development team.\n",
    "\n",
    "While running our tests in this notebook, we'll focus on:\n",
    "\n",
    "- Ensuring that data used for training and testing the model is of appropriate data quality\n",
    "- Ensuring that the raw data has been preprocessed appropriately and that the resulting final datasets reflects this\n",
    "\n",
    "**For a full list of out-of-the-box tests,** refer to our [Test descriptions](https://docs.validmind.ai/developer/model-testing/test-descriptions.html) or try the interactive [Test sandbox](https://docs.validmind.ai/developer/model-testing/test-sandbox.html).\n",
    "\n",
    "<div class=\"alert alert-block alert-info\" style=\"background-color: #B5B5B510; color: black; border: 1px solid #083E44; border-left-width: 5px; box-shadow: 2px 2px 4px rgba(0, 0, 0, 0.2);border-radius: 5px;\"><span style=\"color: #083E44;\"><b>Learn by doing</b></span>\n",
    "<br></br>\n",
    "Our course tailor-made for validators new to ValidMind combines this series of notebooks with more a more in-depth introduction to the ValidMind Platform — <a href=\"https://docs.validmind.ai/training/validator-fundamentals/validator-fundamentals-register.html\" style=\"color: #DE257E;\"><b>Validator Fundamentals</b></a></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "::: {.content-hidden when-format=\"html\"}\n",
    "## Contents    \n",
    "- [Prerequisites](#toc1_)    \n",
    "- [Setting up](#toc2_)    \n",
    "  - [Initialize the ValidMind Library](#toc2_1_)    \n",
    "- [Load the sample dataset](#toc3_)    \n",
    "- [Verifying data quality adjustments](#toc4_)    \n",
    "  - [Identify qualitative tests](#toc4_1_)    \n",
    "  - [Initialize the ValidMind datasets](#toc4_2_)    \n",
    "  - [Run data quality tests](#toc4_3_)    \n",
    "    - [Run tabular data tests](#toc4_3_1_)    \n",
    "  - [Remove highly correlated features](#toc4_4_)    \n",
    "- [Documenting test results](#toc5_)    \n",
    "  - [Configure and run comparison tests](#toc5_1_)    \n",
    "  - [Log tests with unique identifiers](#toc5_2_)    \n",
    "  - [Add test results to reporting](#toc5_3_)    \n",
    "- [Split the preprocessed dataset](#toc6_)    \n",
    "  - [Initialize the split datasets](#toc6_1_)    \n",
    "- [In summary](#toc7_)    \n",
    "- [Next steps](#toc8_)    \n",
    "  - [Develop potential challenger models](#toc8_1_)    \n",
    "\n",
    ":::\n",
    "<!-- jn-toc-notebook-config\n",
    "\tnumbering=false\n",
    "\tanchor=true\n",
    "\tflat=false\n",
    "\tminLevel=2\n",
    "\tmaxLevel=4\n",
    "\t/jn-toc-notebook-config -->\n",
    "<!-- THIS CELL WILL BE REPLACED ON TOC UPDATE. DO NOT WRITE YOUR TEXT IN THIS CELL -->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc1_'></a>\n",
    "\n",
    "## Prerequisites\n",
    "\n",
    "In order to independently assess the quality of your datasets with notebook, you'll need to first have:\n",
    "\n",
    "- [x] Registered a model within the ValidMind Platform and granted yourself access to the model as a validator\n",
    "- [x] Installed the ValidMind Library in your local environment, allowing you to access all its features\n",
    "\n",
    "<div class=\"alert alert-block alert-info\" style=\"background-color: #B5B5B510; color: black; border: 1px solid #083E44; border-left-width: 5px; box-shadow: 2px 2px 4px rgba(0, 0, 0, 0.2);border-radius: 5px;\"><span style=\"color: #083E44;\"><b>Need help with the above steps?</b></span>\n",
    "<br></br>\n",
    "Refer to the first notebook in this series: <a href=\"1-set_up_validmind_for_validation.ipynb\" style=\"color: #DE257E;\"><b>1 — Set up the ValidMind Library for validation</b></a></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc2_'></a>\n",
    "\n",
    "## Setting up"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc2_1_'></a>\n",
    "\n",
    "### Initialize the ValidMind Library\n",
    "\n",
    "First, let's connect up the ValidMind Library to our model we previously registered in the ValidMind Platform:\n",
    "\n",
    "1. In a browser, [log in to ValidMind](https://docs.validmind.ai/guide/configuration/log-in-to-validmind.html).\n",
    "\n",
    "2. In the left sidebar, navigate to **Inventory** and select the model you registered for this \"ValidMind for model validation\" series of notebooks.\n",
    "\n",
    "3. Go to **Getting Started** and click **Copy snippet to clipboard**.\n",
    "\n",
    "Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Make sure the ValidMind Library is installed\n",
    "\n",
    "%pip install -q validmind\n",
    "\n",
    "# Load your model identifier credentials from an `.env` file\n",
    "\n",
    "%load_ext dotenv\n",
    "%dotenv .env\n",
    "\n",
    "# Or replace with your code snippet\n",
    "\n",
    "import validmind as vm\n",
    "\n",
    "vm.init(\n",
    "    # api_host=\"...\",\n",
    "    # api_key=\"...\",\n",
    "    # api_secret=\"...\",\n",
    "    # model=\"...\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc3_'></a>\n",
    "\n",
    "## Load the sample dataset\n",
    "\n",
    "Let's first import the public [Bank Customer Churn Prediction](https://www.kaggle.com/datasets/shantanudhakadd/bank-customer-churn-prediction) dataset from Kaggle, which was used to develop the dummy champion model.\n",
    "\n",
    "We'll use this dataset to review steps that should have been conducted during the initial development and documentation of the model to ensure that the model was built correctly. By independently performing steps taken by the model development team, we can confirm whether the model was built using appropriate and properly processed data.\n",
    "\n",
    "In our below example, note that:\n",
    "\n",
    "- The target column, `Exited` has a value of `1` when a customer has churned and `0` otherwise.\n",
    "- The ValidMind Library provides a wrapper to automatically load the dataset as a [Pandas DataFrame](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.html) object. A Pandas Dataframe is a two-dimensional tabular data structure that makes use of rows and columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from validmind.datasets.classification import customer_churn as demo_dataset\n",
    "\n",
    "print(\n",
    "    f\"Loaded demo dataset with: \\n\\n\\t• Target column: '{demo_dataset.target_column}' \\n\\t• Class labels: {demo_dataset.class_labels}\"\n",
    ")\n",
    "\n",
    "raw_df = demo_dataset.load_data()\n",
    "raw_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc4_'></a>\n",
    "\n",
    "## Verifying data quality adjustments\n",
    "\n",
    "Let's say that thanks to the documentation submitted by the model development team ([Learn more ...](https://docs.validmind.ai/developer/validmind-library.html#for-model-development)), we know that the sample dataset was first modified before being used to train the champion model. After performing some data quality assessments on the raw dataset, it was determined that the dataset required rebalancing, and highly correlated features were also removed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc4_1_'></a>\n",
    "\n",
    "### Identify qualitative tests\n",
    "\n",
    "During model validation, we use the same data processing logic and training procedure to confirm that the model's results can be reproduced independently, so let's start by doing some data quality assessments by running a few individual tests just like the development team did.\n",
    "\n",
    "Use the [`vm.tests.list_tests()` function](https://docs.validmind.ai/validmind/validmind/tests.html#list_tests) introduced by the first notebook in this series in combination with [`vm.tests.list_tags()`](https://docs.validmind.ai/validmind/validmind/tests.html#list_tags) and [`vm.tests.list_tasks()`](https://docs.validmind.ai/validmind/validmind/tests.html#list_tasks) to find which prebuilt tests are relevant for data quality assessment:\n",
    "\n",
    "- **`tasks`** represent the kind of modeling task associated with a test. Here we'll focus on `classification` tasks.\n",
    "- **`tags`** are free-form descriptions providing more details about the test, for example, what category the test falls into. Here we'll focus on the `data_quality` tag."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the list of available task types\n",
    "sorted(vm.tests.list_tasks())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the list of available tags\n",
    "sorted(vm.tests.list_tags())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can pass `tags` and `tasks` as parameters to the `vm.tests.list_tests()` function to filter the tests based on the tags and task types.\n",
    "\n",
    "For example, to find tests related to tabular data quality for classification models, you can call `list_tests()` like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vm.tests.list_tests(task=\"classification\", tags=[\"tabular_data\", \"data_quality\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\" style=\"background-color: #B5B5B510; color: black; border: 1px solid #083E44; border-left-width: 5px; box-shadow: 2px 2px 4px rgba(0, 0, 0, 0.2);border-radius: 5px;\"><span style=\"color: #083E44;\"><b>Want to learn more about navigating ValidMind tests?</b></span>\n",
    "<br></br>\n",
    "Refer to our notebook outlining the utilities available for viewing and understanding available ValidMind tests: <a href=\"https://docs.validmind.ai/notebooks/how_to/explore_tests.html\" style=\"color: #DE257E;\"><b>Explore tests</b></a></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc4_2_'></a>\n",
    "\n",
    "### Initialize the ValidMind datasets\n",
    "\n",
    "With the individual tests we want to run identified, the next step is to connect your data with a ValidMind `Dataset` object. **This step is always necessary every time you want to connect a dataset to documentation and produce test results through ValidMind,** but you only need to do it once per dataset.\n",
    "\n",
    "Initialize a ValidMind dataset object using the [`init_dataset` function](https://docs.validmind.ai/validmind/validmind.html#init_dataset) from the ValidMind (`vm`) module. For this example, we'll pass in the following arguments:\n",
    "\n",
    "- **`dataset`** — The raw dataset that you want to provide as input to tests.\n",
    "- **`input_id`** — A unique identifier that allows tracking what inputs are used when running each individual test.\n",
    "- **`target_column`** — A required argument if tests require access to true values. This is the name of the target column in the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# vm_raw_dataset is now a VMDataset object that you can pass to any ValidMind test\n",
    "vm_raw_dataset = vm.init_dataset(\n",
    "    dataset=raw_df,\n",
    "    input_id=\"raw_dataset\",\n",
    "    target_column=\"Exited\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc4_3_'></a>\n",
    "\n",
    "### Run data quality tests\n",
    "\n",
    "Now that we know how to initialize a ValidMind `dataset` object, we're ready to run some tests!\n",
    "\n",
    "You run individual tests by calling [the `run_test` function](https://docs.validmind.ai/validmind/validmind/tests.html#run_test) provided by the `validmind.tests` module. For the examples below, we'll pass in the following arguments:\n",
    "\n",
    "- **`test_id`** — The ID of the test to run, as seen in the `ID` column when you run `list_tests`. \n",
    "- **`params`** — A dictionary of parameters for the test. These will override any `default_params` set in the test definition. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc4_3_1_'></a>\n",
    "\n",
    "#### Run tabular data tests\n",
    "\n",
    "The inputs expected by a test can also be found in the test definition — let's take [`validmind.data_validation.DescriptiveStatistics`](https://docs.validmind.ai/tests/data_validation/DescriptiveStatistics.html) as an example.\n",
    "\n",
    "Note that the output of the [`describe_test()` function](https://docs.validmind.ai/validmind/validmind/tests.html#describe_test) below shows that this test expects a `dataset` as input:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vm.tests.describe_test(\"validmind.data_validation.DescriptiveStatistics\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's run a few tests to assess the quality of the dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "result2 = vm.tests.run_test(\n",
    "    test_id=\"validmind.data_validation.ClassImbalance\",\n",
    "    inputs={\"dataset\": vm_raw_dataset},\n",
    "    params={\"min_percent_threshold\": 30},\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The output above shows that [the class imbalance test](https://docs.validmind.ai/tests/data_validation/ClassImbalance.html) did not pass according to the value we set for `min_percent_threshold` — great, this matches what was reported by the model development team.\n",
    "\n",
    "To address this issue, we'll re-run the test on some processed data. In this case let's apply a very simple rebalancing technique to the dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "raw_copy_df = raw_df.sample(frac=1)  # Create a copy of the raw dataset\n",
    "\n",
    "# Create a balanced dataset with the same number of exited and not exited customers\n",
    "exited_df = raw_copy_df.loc[raw_copy_df[\"Exited\"] == 1]\n",
    "not_exited_df = raw_copy_df.loc[raw_copy_df[\"Exited\"] == 0].sample(n=exited_df.shape[0])\n",
    "\n",
    "balanced_raw_df = pd.concat([exited_df, not_exited_df])\n",
    "balanced_raw_df = balanced_raw_df.sample(frac=1, random_state=42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With this new balanced dataset, you can re-run the individual test to see if it now passes the class imbalance test requirement.\n",
    "\n",
    "As this is technically a different dataset, **remember to first initialize a new ValidMind `Dataset` object** to pass in as input as required by `run_test()`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Register new data and now 'balanced_raw_dataset' is the new dataset object of interest\n",
    "vm_balanced_raw_dataset = vm.init_dataset(\n",
    "    dataset=balanced_raw_df,\n",
    "    input_id=\"balanced_raw_dataset\",\n",
    "    target_column=\"Exited\",\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Pass the initialized `balanced_raw_dataset` as input into the test run\n",
    "result = vm.tests.run_test(\n",
    "    test_id=\"validmind.data_validation.ClassImbalance\",\n",
    "    inputs={\"dataset\": vm_balanced_raw_dataset},\n",
    "    params={\"min_percent_threshold\": 30},\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc4_4_'></a>\n",
    "\n",
    "### Remove highly correlated features\n",
    "\n",
    "Next, let's also remove highly correlated features from our dataset as outlined by the development team. Removing highly correlated features helps make the model simpler, more stable, and easier to understand.\n",
    "\n",
    "You can utilize the output from a ValidMind test for further use — in this below example, to retrieve the list of features with the highest correlation coefficients and use them to reduce the final list of features for modeling.\n",
    "\n",
    "First, we'll run [`validmind.data_validation.HighPearsonCorrelation`](https://docs.validmind.ai/tests/data_validation/HighPearsonCorrelation.html) with the `balanced_raw_dataset` we initialized previously as input as is for comparison with later runs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "corr_result = vm.tests.run_test(\n",
    "    test_id=\"validmind.data_validation.HighPearsonCorrelation\",\n",
    "    params={\"max_threshold\": 0.3},\n",
    "    inputs={\"dataset\": vm_balanced_raw_dataset},\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The output above shows that the test did not pass according to the value we set for `max_threshold` — as reported and expected.\n",
    "\n",
    "`corr_result` is an object of type `TestResult`. We can inspect the result object to see what the test has produced:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(type(corr_result))\n",
    "print(\"Result ID: \", corr_result.result_id)\n",
    "print(\"Params: \", corr_result.params)\n",
    "print(\"Passed: \", corr_result.passed)\n",
    "print(\"Tables: \", corr_result.tables)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's remove the highly correlated features and create a new VM `dataset` object.\n",
    "\n",
    "We'll begin by checking out the table in the result and extracting a list of features that failed the test:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Extract table from `corr_result.tables`\n",
    "features_df = corr_result.tables[0].data\n",
    "features_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Extract list of features that failed the test\n",
    "high_correlation_features = features_df[features_df[\"Pass/Fail\"] == \"Fail\"][\"Columns\"].tolist()\n",
    "high_correlation_features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, extract the feature names from the list of strings (example: `(Age, Exited)` > `Age`):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "high_correlation_features = [feature.split(\",\")[0].strip(\"()\") for feature in high_correlation_features]\n",
    "high_correlation_features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, it's time to re-initialize the dataset with the highly correlated features removed.\n",
    "\n",
    "**Note the use of a different `input_id`.** This allows tracking the inputs used when running each individual test."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Remove the highly correlated features from the dataset\n",
    "balanced_raw_no_age_df = balanced_raw_df.drop(columns=high_correlation_features)\n",
    "\n",
    "# Re-initialize the dataset object\n",
    "vm_raw_dataset_preprocessed = vm.init_dataset(\n",
    "    dataset=balanced_raw_no_age_df,\n",
    "    input_id=\"raw_dataset_preprocessed\",\n",
    "    target_column=\"Exited\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Re-running the test with the reduced feature set should pass the test:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "corr_result = vm.tests.run_test(\n",
    "    test_id=\"validmind.data_validation.HighPearsonCorrelation\",\n",
    "    params={\"max_threshold\": 0.3},\n",
    "    inputs={\"dataset\": vm_raw_dataset_preprocessed},\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also plot the correlation matrix to visualize the new correlation between features:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "corr_result = vm.tests.run_test(\n",
    "    test_id=\"validmind.data_validation.PearsonCorrelationMatrix\",\n",
    "    inputs={\"dataset\": vm_raw_dataset_preprocessed},\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc5_'></a>\n",
    "\n",
    "## Documenting test results\n",
    "\n",
    "Now that we've done some analysis on two different datasets, we can use ValidMind to easily document why certain things were done to our raw data with testing to support it. Every test result returned by the `run_test()` function has a `.log()` method that can be used to send the test results to the ValidMind Platform.\n",
    "\n",
    "When logging validation test results to the platform, you'll need to manually add those results to the desired section of the validation report. To demonstrate how to add test results to your validation report, we'll log our data quality tests and insert the results via the ValidMind Platform."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc5_1_'></a>\n",
    "\n",
    "### Configure and run comparison tests\n",
    "\n",
    "Below, we'll perform comparison tests between the original raw dataset (`raw_dataset`) and the final preprocessed (`raw_dataset_preprocessed`) dataset, again logging the results to the ValidMind Platform. \n",
    "\n",
    "We can specify all the tests we'd ike to run in a dictionary called `test_config`, and we'll pass in the following arguments for each test:\n",
    "\n",
    "  - **`params`:** Individual test parameters.\n",
    "  - **`input_grid`:** Individual test inputs to compare. In this case, we'll input our two datasets for comparison.\n",
    "\n",
    "**Note here that the `input_grid` expects the `input_id` of the dataset as the value rather than the variable name we specified:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Individual test config with inputs specified\n",
    "test_config = {\n",
    "    \"validmind.data_validation.ClassImbalance\": {\n",
    "        \"input_grid\": {\"dataset\": [\"raw_dataset\", \"raw_dataset_preprocessed\"]},\n",
    "        \"params\": {\"min_percent_threshold\": 30}\n",
    "    },\n",
    "    \"validmind.data_validation.HighPearsonCorrelation\": {\n",
    "        \"input_grid\": {\"dataset\": [\"raw_dataset\", \"raw_dataset_preprocessed\"]},\n",
    "        \"params\": {\"max_threshold\": 0.3}\n",
    "    },\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then batch run and log our tests in `test_config`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for t in test_config:\n",
    "    print(t)\n",
    "    try:\n",
    "        # Check if test has input_grid\n",
    "        if 'input_grid' in test_config[t]:\n",
    "            # For tests with input_grid, pass the input_grid configuration\n",
    "            if 'params' in test_config[t]:\n",
    "                vm.tests.run_test(t, input_grid=test_config[t]['input_grid'], params=test_config[t]['params']).log()\n",
    "            else:\n",
    "                vm.tests.run_test(t, input_grid=test_config[t]['input_grid']).log()\n",
    "        else:\n",
    "            # Original logic for regular inputs\n",
    "            if 'params' in test_config[t]:\n",
    "                vm.tests.run_test(t, inputs=test_config[t]['inputs'], params=test_config[t]['params']).log()\n",
    "            else:\n",
    "                vm.tests.run_test(t, inputs=test_config[t]['inputs']).log()\n",
    "    except Exception as e:\n",
    "        print(f\"Error running test {t}: {str(e)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\" style=\"background-color: #B5B5B510; color: black; border: 1px solid #083E44; border-left-width: 5px; box-shadow: 2px 2px 4px rgba(0, 0, 0, 0.2);border-radius: 5px;\"><span style=\"color: #083E44;\"><b>Note the output returned indicating that a test-driven block doesn't currently exist in your model's documentation for some test IDs. </b></span>\n",
    "<br></br>\n",
    "That's expected, as when we run validations tests the results logged need to be manually added to your report as part of your compliance assessment process within the ValidMind Platform.</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc5_2_'></a>\n",
    "\n",
    "### Log tests with unique identifiers\n",
    "\n",
    "Next, we'll use the previously initialized `vm_balanced_raw_dataset` (that still has a highly correlated `Age` column) as input to run an individual test, then log the result to the ValidMind Platform.\n",
    "\n",
    "When running individual tests, **you can use a custom `result_id` to tag the individual result with a unique identifier:**\n",
    "\n",
    "- This `result_id` can be appended to `test_id` with a `:` separator.\n",
    "- The `balanced_raw_dataset` result identifier will correspond to the `balanced_raw_dataset` input, the dataset that still has the `Age` column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "result = vm.tests.run_test(\n",
    "    test_id=\"validmind.data_validation.HighPearsonCorrelation:balanced_raw_dataset\",\n",
    "    params={\"max_threshold\": 0.3},\n",
    "    inputs={\"dataset\": vm_balanced_raw_dataset},\n",
    ")\n",
    "result.log()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc5_3_'></a>\n",
    "\n",
    "### Add test results to reporting\n",
    "\n",
    "With some test results logged, let's head to the model we connected to at the beginning of this notebook and learn how to insert a test result into our validation report ([Need more help?](https://docs.validmind.ai/guide/model-validation/assess-compliance.html#link-validator-evidence)).\n",
    "\n",
    "While the example below focuses on a specific test result, you can follow the same general procedure for your other results:\n",
    "\n",
    "1. From the **Inventory** in the ValidMind Platform, go to the model you connected to earlier.\n",
    "\n",
    "2. In the left sidebar that appears for your model, click **Validation Report**.\n",
    "\n",
    "3. Locate the Data Preparation section and click on **2.2.1. Data Quality** to expand that section.\n",
    "\n",
    "4. Under the Class Imbalance Assessment section, locate Validator Evidence then click **Link Evidence to Report**:\n",
    "\n",
    "    <img src= \"link-validator-evidence.png\" alt=\"Screenshot showing the validation report with the link validator evidence to report option highlighted\" style=\"border: 2px solid #083E44; border-radius: 8px; border-right-width: 2px; border-bottom-width: 3px;\">\n",
    "    <br><br>\n",
    "\n",
    "5. Select the Class Imbalance test results we logged: **ValidMind Data Validation Class Imbalance** \n",
    "\n",
    "    <img src= \"selecting-class-imbalance-results.png\" alt=\"Screenshot showing the ClassImbalance test selected\" style=\"border: 2px solid #083E44; border-radius: 8px; border-right-width: 2px; border-bottom-width: 3px;\">\n",
    "    <br><br>\n",
    "\n",
    "6. Click **Update Linked Evidence** to add the test results to the validation report.\n",
    "\n",
    "    Confirm that the results for the Class Imbalance test you inserted has been correctly inserted into section **2.2.1. Data Quality** of the report:\n",
    "\n",
    "    <img src= \"inserted-class-imbalance-results.png\" alt=\"Screenshot showing the ClassImbalance test inserted into the validation report\" style=\"border: 2px solid #083E44; border-radius: 8px; border-right-width: 2px; border-bottom-width: 3px;\">\n",
    "    <br><br>\n",
    "\n",
    "7. Note that these test results are flagged as **Requires Attention** — as they include comparative results from our initial raw dataset.\n",
    "\n",
    "    Click **See evidence details** to review the LLM-generated description that summarizes the test results, that confirm that our final preprocessed dataset actually passes our test:\n",
    "\n",
    "    <img src= \"class-imbalance-results-detail.png\" alt=\"Screenshot showing the ClassImbalance test generated description in the text editor\" style=\"border: 2px solid #083E44; border-radius: 8px; border-right-width: 2px; border-bottom-width: 3px;\">\n",
    "    <br><br>\n",
    "\n",
    "\n",
    "<div class=\"alert alert-block alert-info\" style=\"background-color: #B5B5B510; color: black; border: 1px solid #083E44; border-left-width: 5px; box-shadow: 2px 2px 4px rgba(0, 0, 0, 0.2);border-radius: 5px;\"><span style=\"color: #083E44;\"><b>    Here in this text editor, you can make qualitative edits to the draft that ValidMind generated to finalize the test results.</b></span>\n",
    "<br></br>\n",
    "Learn more: <a href=\"https://docs.validmind.ai/guide/model-documentation/work-with-content-blocks.html\" style=\"color: #DE257E;\"><b>Work with content blocks</b></a></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc6_'></a>\n",
    "\n",
    "## Split the preprocessed dataset\n",
    "\n",
    "With our raw dataset rebalanced with highly correlated features removed, let's now **spilt our dataset into train and test** in preparation for model evaluation testing.\n",
    "\n",
    "To start, let's grab the first few rows from the `balanced_raw_no_age_df` dataset we initialized earlier:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "balanced_raw_no_age_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before training the model, we need to encode the categorical features in the dataset:\n",
    "\n",
    "- Use the `OneHotEncoder` class from the `sklearn.preprocessing` module to encode the categorical features.\n",
    "- The categorical features in the dataset are `Geography` and `Gender`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "balanced_raw_no_age_df = pd.get_dummies(\n",
    "    balanced_raw_no_age_df, columns=[\"Geography\", \"Gender\"], drop_first=True\n",
    ")\n",
    "balanced_raw_no_age_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Splitting our dataset into training and testing is essential for proper validation testing, as this helps assess how well the model generalizes to unseen data:\n",
    "\n",
    "- We start by dividing our `balanced_raw_no_age_df` dataset into training and test subsets using `train_test_split`, with 80% of the data allocated to training (`train_df`) and 20% to testing (`test_df`).\n",
    "- From each subset, we separate the features (all columns except \"Exited\") into `X_train` and `X_test`, and the target column (\"Exited\") into `y_train` and `y_test`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "train_df, test_df = train_test_split(balanced_raw_no_age_df, test_size=0.20)\n",
    "\n",
    "X_train = train_df.drop(\"Exited\", axis=1)\n",
    "y_train = train_df[\"Exited\"]\n",
    "X_test = test_df.drop(\"Exited\", axis=1)\n",
    "y_test = test_df[\"Exited\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc6_1_'></a>\n",
    "\n",
    "### Initialize the split datasets\n",
    "\n",
    "Next, let's initialize the training and testing datasets so they are available for use:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vm_train_ds = vm.init_dataset(\n",
    "    input_id=\"train_dataset_final\",\n",
    "    dataset=train_df,\n",
    "    target_column=\"Exited\",\n",
    ")\n",
    "\n",
    "vm_test_ds = vm.init_dataset(\n",
    "    input_id=\"test_dataset_final\",\n",
    "    dataset=test_df,\n",
    "    target_column=\"Exited\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc7_'></a>\n",
    "\n",
    "## In summary\n",
    "\n",
    "In this second notebook, you learned how to:\n",
    "\n",
    "- [x] Import a sample dataset\n",
    "- [x] Identify which tests you might want to run with ValidMind\n",
    "- [x] Initialize ValidMind datasets\n",
    "- [x] Run individual tests\n",
    "- [x] Utilize the output from tests you’ve run\n",
    "- [x] Log test results as evidence to the ValidMind Platform\n",
    "- [x] Insert test results into your validation report"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc8_'></a>\n",
    "\n",
    "## Next steps\n",
    "\n",
    "<a id='toc8_1_'></a>\n",
    "\n",
    "### Develop potential challenger models\n",
    "\n",
    "Now that you're familiar with the basics of using the ValidMind Library, let's use it to develop a challenger model: **[3 — Developing a potential challenger model](3-developing_challenger_model.ipynb)**"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "ValidMind Library",
   "language": "python",
   "name": "validmind"
  },
  "language_info": {
   "name": "python",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
