{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# ValidMind for model validation 3 — Developing a potential challenger model\n",
    "\n",
    "Learn how to use ValidMind for your end-to-end model validation process with our series of four introductory notebooks. In this third notebook, develop a potential challenger model and then pass your model and its predictions to ValidMind.\n",
    "\n",
    "A *challenger model* is an alternate model that attempts to outperform the champion model, ensuring that the best performing fit-for-purpose model is always considered for deployment. Challenger models also help avoid over-reliance on a single model, and allow testing of new features, algorithms, or data sources without disrupting the production lifecycle.\n",
    "\n",
    "<div class=\"alert alert-block alert-info\" style=\"background-color: #B5B5B510; color: black; border: 1px solid #083E44; border-left-width: 5px; box-shadow: 2px 2px 4px rgba(0, 0, 0, 0.2);border-radius: 5px;\"><span style=\"color: #083E44;\"><b>Learn by doing</b></span>\n",
    "<br></br>\n",
    "Our course tailor-made for validators new to ValidMind combines this series of notebooks with more a more in-depth introduction to the ValidMind Platform — <a href=\"https://docs.validmind.ai/training/validator-fundamentals/validator-fundamentals-register.html\" style=\"color: #DE257E;\"><b>Validator Fundamentals</b></a></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "::: {.content-hidden when-format=\"html\"}\n",
    "## Contents    \n",
    "- [Prerequisites](#toc1_)    \n",
    "- [Setting up](#toc2_)    \n",
    "  - [Initialize the ValidMind Library](#toc2_1_)    \n",
    "  - [Import the sample dataset](#toc2_2_)    \n",
    "    - [Preprocess the dataset](#toc2_2_1_)    \n",
    "  - [Split the preprocessed dataset](#toc2_3_)    \n",
    "- [Import the champion model](#toc3_)    \n",
    "- [Training a potential challenger model](#toc4_)    \n",
    "  - [Random forest classification model](#toc4_1_)    \n",
    "- [Initializing the model objects](#toc5_)    \n",
    "  - [Initialize the model objects](#toc5_1_)    \n",
    "  - [Assign predictions](#toc5_2_)    \n",
    "- [Running model evaluation tests](#toc6_)    \n",
    "  - [Run model performance tests](#toc6_1_)    \n",
    "    - [Evaluate performance of the champion model](#toc6_1_1_)    \n",
    "    - [Log a model finding](#toc6_1_2_)    \n",
    "    - [Evaluate performance of challenger model](#toc6_1_3_)    \n",
    "  - [Run diagnostic tests](#toc6_2_)    \n",
    "  - [Run feature importance tests](#toc6_3_)    \n",
    "- [In summary](#toc7_)    \n",
    "- [Next steps](#toc8_)    \n",
    "  - [Finalize validation and reporting](#toc8_1_)    \n",
    "\n",
    ":::\n",
    "<!-- jn-toc-notebook-config\n",
    "\tnumbering=false\n",
    "\tanchor=true\n",
    "\tflat=false\n",
    "\tminLevel=2\n",
    "\tmaxLevel=4\n",
    "\t/jn-toc-notebook-config -->\n",
    "<!-- THIS CELL WILL BE REPLACED ON TOC UPDATE. DO NOT WRITE YOUR TEXT IN THIS CELL -->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc1_'></a>\n",
    "\n",
    "## Prerequisites\n",
    "\n",
    "In order to develop potential challenger models with this notebook, you'll need to first have:\n",
    "\n",
    "- [x] Registered a model within the ValidMind Platform and granted yourself access to the model as a validator\n",
    "- [x] Installed the ValidMind Library in your local environment, allowing you to access all its features\n",
    "- [x] Learned how to import and initialize datasets for use with ValidMind\n",
    "- [x] Understood the basics of how to run and log tests with ValidMind\n",
    "- [x] Run data quality tests on the datasets used to train the champion model, and logged the results of those tests to ValidMind\n",
    "- [x] Inserted your logged test results into your validation report\n",
    "\n",
    "<div class=\"alert alert-block alert-info\" style=\"background-color: #B5B5B510; color: black; border: 1px solid #083E44; border-left-width: 5px; box-shadow: 2px 2px 4px rgba(0, 0, 0, 0.2);border-radius: 5px;\"><span style=\"color: #083E44;\"><b>Need help with the above steps?</b></span>\n",
    "<br></br>\n",
    "Refer to the first two notebooks in this series:\n",
    "\n",
    "- <a href=\"1-set_up_validmind_for_validation.ipynb\" style=\"color: #DE257E;\"><b>1 — Set up the ValidMind Library for validation</b></a>\n",
    "- <a href=\"2-start_validation_process.ipynb\" style=\"color: #DE257E;\"><b>2 — Start the model validation process</b></a>\n",
    "\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc2_'></a>\n",
    "\n",
    "## Setting up\n",
    "\n",
    "This section should be quite familiar to you — as we performed the same actions in the previous notebook, **[2 — Start the model validation process](2-start_validation_process.ipynb)**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc2_1_'></a>\n",
    "\n",
    "### Initialize the ValidMind Library\n",
    "\n",
    "As usual, let's first connect up the ValidMind Library to our model we previously registered in the ValidMind Platform:\n",
    "\n",
    "1. In a browser, [log in to ValidMind](https://docs.validmind.ai/guide/configuration/log-in-to-validmind.html).\n",
    "\n",
    "2. In the left sidebar, navigate to **Inventory** and select the model you registered for this \"ValidMind for model validation\" series of notebooks.\n",
    "\n",
    "3. Go to **Getting Started** and click **Copy snippet to clipboard**.\n",
    "\n",
    "Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Make sure the ValidMind Library is installed\n",
    "\n",
    "%pip install -q validmind\n",
    "\n",
    "# Load your model identifier credentials from an `.env` file\n",
    "\n",
    "%load_ext dotenv\n",
    "%dotenv .env\n",
    "\n",
    "# Or replace with your code snippet\n",
    "\n",
    "import validmind as vm\n",
    "\n",
    "vm.init(\n",
    "    # api_host=\"...\",\n",
    "    # api_key=\"...\",\n",
    "    # api_secret=\"...\",\n",
    "    # model=\"...\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc2_2_'></a>\n",
    "\n",
    "### Import the sample dataset\n",
    "\n",
    "Next, we'll load in the sample [Bank Customer Churn Prediction](https://www.kaggle.com/datasets/shantanudhakadd/bank-customer-churn-prediction) dataset used to develop the champion model that we will independently preprocess:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load the sample dataset\n",
    "from validmind.datasets.classification import customer_churn as demo_dataset\n",
    "\n",
    "print(\n",
    "    f\"Loaded demo dataset with: \\n\\n\\t• Target column: '{demo_dataset.target_column}' \\n\\t• Class labels: {demo_dataset.class_labels}\"\n",
    ")\n",
    "\n",
    "raw_df = demo_dataset.load_data()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc2_2_1_'></a>\n",
    "\n",
    "#### Preprocess the dataset\n",
    "\n",
    "We’ll apply a simple rebalancing technique to the dataset before continuing:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "raw_copy_df = raw_df.sample(frac=1)  # Create a copy of the raw dataset\n",
    "\n",
    "# Create a balanced dataset with the same number of exited and not exited customers\n",
    "exited_df = raw_copy_df.loc[raw_copy_df[\"Exited\"] == 1]\n",
    "not_exited_df = raw_copy_df.loc[raw_copy_df[\"Exited\"] == 0].sample(n=exited_df.shape[0])\n",
    "\n",
    "balanced_raw_df = pd.concat([exited_df, not_exited_df])\n",
    "balanced_raw_df = balanced_raw_df.sample(frac=1, random_state=42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let’s also quickly remove highly correlated features from the dataset using the output from a ValidMind test.\n",
    "\n",
    "As you know, before we can run tests you’ll need to initialize a ValidMind dataset object with the [`init_dataset` function](https://docs.validmind.ai/validmind/validmind.html#init_dataset):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Register new data and now 'balanced_raw_dataset' is the new dataset object of interest\n",
    "vm_balanced_raw_dataset = vm.init_dataset(\n",
    "    dataset=balanced_raw_df,\n",
    "    input_id=\"balanced_raw_dataset\",\n",
    "    target_column=\"Exited\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With our balanced dataset initialized, we can then run our test and utilize the output to help us identify the features we want to remove:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run HighPearsonCorrelation test with our balanced dataset as input and return a result object\n",
    "corr_result = vm.tests.run_test(\n",
    "    test_id=\"validmind.data_validation.HighPearsonCorrelation\",\n",
    "    params={\"max_threshold\": 0.3},\n",
    "    inputs={\"dataset\": vm_balanced_raw_dataset},\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# From result object, extract table from `corr_result.tables`\n",
    "features_df = corr_result.tables[0].data\n",
    "features_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Extract list of features that failed the test\n",
    "high_correlation_features = features_df[features_df[\"Pass/Fail\"] == \"Fail\"][\"Columns\"].tolist()\n",
    "high_correlation_features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Extract feature names from the list of strings\n",
    "high_correlation_features = [feature.split(\",\")[0].strip(\"()\") for feature in high_correlation_features]\n",
    "high_correlation_features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can then re-initialize the dataset with a different `input_id` and the highly correlated features removed and re-run the test for confirmation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Remove the highly correlated features from the dataset\n",
    "balanced_raw_no_age_df = balanced_raw_df.drop(columns=high_correlation_features)\n",
    "\n",
    "# Re-initialize the dataset object\n",
    "vm_raw_dataset_preprocessed = vm.init_dataset(\n",
    "    dataset=balanced_raw_no_age_df,\n",
    "    input_id=\"raw_dataset_preprocessed\",\n",
    "    target_column=\"Exited\",\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Re-run the test with the reduced feature set\n",
    "corr_result = vm.tests.run_test(\n",
    "    test_id=\"validmind.data_validation.HighPearsonCorrelation\",\n",
    "    params={\"max_threshold\": 0.3},\n",
    "    inputs={\"dataset\": vm_raw_dataset_preprocessed},\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc2_3_'></a>\n",
    "\n",
    "### Split the preprocessed dataset\n",
    "\n",
    "With our raw dataset rebalanced with highly correlated features removed, let's now **spilt our dataset into train and test** in preparation for model evaluation testing:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Encode categorical features in the dataset\n",
    "balanced_raw_no_age_df = pd.get_dummies(\n",
    "    balanced_raw_no_age_df, columns=[\"Geography\", \"Gender\"], drop_first=True\n",
    ")\n",
    "balanced_raw_no_age_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "# Split the dataset into train and test\n",
    "train_df, test_df = train_test_split(balanced_raw_no_age_df, test_size=0.20)\n",
    "\n",
    "X_train = train_df.drop(\"Exited\", axis=1)\n",
    "y_train = train_df[\"Exited\"]\n",
    "X_test = test_df.drop(\"Exited\", axis=1)\n",
    "y_test = test_df[\"Exited\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize the split datasets\n",
    "vm_train_ds = vm.init_dataset(\n",
    "    input_id=\"train_dataset_final\",\n",
    "    dataset=train_df,\n",
    "    target_column=\"Exited\",\n",
    ")\n",
    "\n",
    "vm_test_ds = vm.init_dataset(\n",
    "    input_id=\"test_dataset_final\",\n",
    "    dataset=test_df,\n",
    "    target_column=\"Exited\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc3_'></a>\n",
    "\n",
    "## Import the champion model\n",
    "\n",
    "With our raw dataset assessed and preprocessed, let's go ahead and import the champion model submitted by the model development team in the format of a `.pkl` file: **[lr_model_champion.pkl](lr_model_champion.pkl)**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import the champion model\n",
    "import pickle as pkl\n",
    "\n",
    "with open(\"lr_model_champion.pkl\", \"rb\") as f:\n",
    "    log_reg = pkl.load(f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc4_'></a>\n",
    "\n",
    "## Training a potential challenger model\n",
    "\n",
    "We're curious how an alternate model compares to our champion model, so let's train a challenger model as a basis for our testing.\n",
    "\n",
    "Our champion *logistic regression model* is a simpler, parametric model that assumes a linear relationship between the independent variables and the log-odds of the outcome. While logistic regression may not capture complex patterns as effectively, it offers a high degree of interpretability and is easier to explain to stakeholders. However, model risk is not calculated in isolation from a single factor, but rather in consideration with trade-offs in predictive performance, ease of interpretability, and overall alignment with business objectives."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc4_1_'></a>\n",
    "\n",
    "### Random forest classification model\n",
    "\n",
    "A *random forest classification model* is an ensemble machine learning algorithm that uses multiple decision trees to classify data. In ensemble learning, multiple models are combined to improve prediction accuracy and robustness.\n",
    "\n",
    "Random forest classification models generally have higher accuracy because they capture complex, non-linear relationships, but as a result they lack transparency in their predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import the Random Forest Classification model\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "\n",
    "# Create the model instance with 50 decision trees\n",
    "rf_model = RandomForestClassifier(\n",
    "    n_estimators=50,\n",
    "    random_state=42,\n",
    ")\n",
    "\n",
    "# Train the model\n",
    "rf_model.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc5_'></a>\n",
    "\n",
    "## Initializing the model objects"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc5_1_'></a>\n",
    "\n",
    "### Initialize the model objects\n",
    "\n",
    "In addition to the initialized datasets, you'll also need to initialize a ValidMind model object (`vm_model`) that can be passed to other functions for analysis and tests on the data for each of our two models.\n",
    "\n",
    "You simply initialize this model object with [`vm.init_model()`](https://docs.validmind.ai/validmind/validmind.html#init_model):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize the champion logistic regression model\n",
    "vm_log_model = vm.init_model(\n",
    "    log_reg,\n",
    "    input_id=\"log_model_champion\",\n",
    ")\n",
    "\n",
    "# Initialize the challenger random forest classification model\n",
    "vm_rf_model = vm.init_model(\n",
    "    rf_model,\n",
    "    input_id=\"rf_model\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc5_2_'></a>\n",
    "\n",
    "### Assign predictions\n",
    "\n",
    "With our models registered, we'll move on to assigning both the predictive probabilities coming directly from each model's predictions, and the binary prediction after applying the cutoff threshold described in the Compute binary predictions step above.\n",
    "\n",
    "- The [`assign_predictions()` method](https://docs.validmind.ai/validmind/validmind/vm_models.html#assign_predictions) from the `Dataset` object can link existing predictions to any number of models.\n",
    "- This method links the model's class prediction values and probabilities to our `vm_train_ds` and `vm_test_ds` datasets.\n",
    "\n",
    "If no prediction values are passed, the method will compute predictions automatically:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Champion — Logistic regression model\n",
    "vm_train_ds.assign_predictions(model=vm_log_model)\n",
    "vm_test_ds.assign_predictions(model=vm_log_model)\n",
    "\n",
    "# Challenger — Random forest classification model\n",
    "vm_train_ds.assign_predictions(model=vm_rf_model)\n",
    "vm_test_ds.assign_predictions(model=vm_rf_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc6_'></a>\n",
    "\n",
    "## Running model evaluation tests\n",
    "\n",
    "With our setup complete, let's run the rest of our validation tests. Since we have already verified the data quality of the dataset used to train our champion model, we will now focus on comprehensive performance evaluations of both the champion and challenger models."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc6_1_'></a>\n",
    "\n",
    "### Run model performance tests\n",
    "\n",
    "Let's run some performance tests, beginning with independent testing of our champion logistic regression model, then moving on to our potential challenger model.\n",
    "\n",
    "Use [`vm.tests.list_tests()`](https://docs.validmind.ai/validmind/validmind/tests.html#list_tests) to identify all the model performance tests for classification:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "vm.tests.list_tests(tags=[\"model_performance\"], task=\"classification\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll isolate the specific tests we want to run in `mpt`:\n",
    "\n",
    "- [`ClassifierPerformance`](https://docs.validmind.ai/tests/model_validation/sklearn/ClassifierPerformance.html)\n",
    "- [`ConfusionMatrix`](https://docs.validmind.ai/tests/model_validation/sklearn/ConfusionMatrix.html)\n",
    "- [`MinimumAccuracy`](https://docs.validmind.ai/tests/model_validation/sklearn/MinimumAccuracy.html)\n",
    "- [`MinimumF1Score`](https://docs.validmind.ai/tests/model_validation/sklearn/MinimumF1Score.html)\n",
    "- [`ROCCurve`](https://docs.validmind.ai/tests/model_validation/sklearn/ROCCurve.html)\n",
    "\n",
    "As we learned in the previous notebook [2 — Start the model validation process](2-start_validation_process.ipynb), you can use a custom `result_id` to tag the individual result with a unique identifier by appending this `result_id` to the `test_id` with a `:` separator. We'll append an identifier for our champion model here:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mpt = [\n",
    "    \"validmind.model_validation.sklearn.ClassifierPerformance:logreg_champion\",\n",
    "    \"validmind.model_validation.sklearn.ConfusionMatrix:logreg_champion\",\n",
    "    \"validmind.model_validation.sklearn.MinimumAccuracy:logreg_champion\",\n",
    "    \"validmind.model_validation.sklearn.MinimumF1Score:logreg_champion\",\n",
    "    \"validmind.model_validation.sklearn.ROCCurve:logreg_champion\"\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc6_1_1_'></a>\n",
    "\n",
    "#### Evaluate performance of the champion model\n",
    "\n",
    "Now, let's run and log our batch of model performance tests using our testing dataset (`vm_test_ds`) for our champion model:\n",
    "\n",
    "- The test set serves as a proxy for real-world data, providing an unbiased estimate of model performance since it was not used during training or tuning.\n",
    "- The test set also acts as protection against selection bias and model tweaking, giving a final, more unbiased checkpoint."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for test in mpt:\n",
    "    vm.tests.run_test(\n",
    "        test,\n",
    "        inputs={\n",
    "            \"dataset\": vm_test_ds, \"model\" : vm_log_model,\n",
    "        },\n",
    "    ).log()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\" style=\"background-color: #B5B5B510; color: black; border: 1px solid #083E44; border-left-width: 5px; box-shadow: 2px 2px 4px rgba(0, 0, 0, 0.2);border-radius: 5px;\"><span style=\"color: #083E44;\"><b>Note the output returned indicating that a test-driven block doesn't currently exist in your model's documentation for some test IDs. </b></span>\n",
    "<br></br>\n",
    "That's expected, as when we run validations tests the results logged need to be manually added to your report as part of your compliance assessment process within the ValidMind Platform.</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc6_1_2_'></a>\n",
    "\n",
    "#### Log a model finding\n",
    "\n",
    "As we can observe from the output above, our champion model doesn't pass the `MinimumAccuracy` based on the default thresholds of the out-of-the-box test, so let's log a model finding in the ValidMind Platform  ([Need more help?](https://docs.validmind.ai/guide/model-validation/add-manage-model-findings.html)):\n",
    "\n",
    "1. From the **Inventory** in the ValidMind Platform, go to the model you connected to earlier.\n",
    "\n",
    "2. In the left sidebar that appears for your model, click **Validation Report**.\n",
    "\n",
    "3. Locate the Data Preparation section and click on **2.2.2. Model Performance** to expand that section.\n",
    "\n",
    "4. Under the Model Performance Metrics section, locate Findings then click **Link Finding to Report**:\n",
    "\n",
    "    <img src= \"link-finding.png\" alt=\"Screenshot showing the validation report with the link finding option highlighted\" style=\"border: 2px solid #083E44; border-radius: 8px; border-right-width: 2px; border-bottom-width: 3px;\">\n",
    "    <br><br>\n",
    "\n",
    "5. Click **+ Create New Finding** to add a finding.\n",
    "\n",
    "6. Enter in the details for your finding, for example:\n",
    "\n",
    "    - **TITLE** — Champion Logistic Regression Model Fails Minimum Accuracy Threshold\n",
    "    - **RISK AREA** — Model Performance\n",
    "    - **DOCUMENTATION SECTION** — 3.2. Model Evaluation\n",
    "    - **DESCRIPTION** — The logistic regression champion model was subjected to a Minimum Accuracy test to determine whether its predictive accuracy meets the predefined performance threshold of 0.7. The model achieved an accuracy score of 0.6136, which falls below the required minimum. As a result, the test produced a Fail outcome.\n",
    "\n",
    "7. Click **Save**.\n",
    "\n",
    "8. Select the finding you just added to link to your validation report:\n",
    "\n",
    "    <img src= \"select-finding.png\" alt=\"Screenshot showing the newly created finding selected to be inserted\" style=\"border: 2px solid #083E44; border-radius: 8px; border-right-width: 2px; border-bottom-width: 3px;\">\n",
    "    <br><br>\n",
    "\n",
    "9. Click **Update Linked Findings** to insert your finding.\n",
    "\n",
    "10. Confirm that finding you inserted has been correctly inserted into section **2.2.2. Model Performance**  of the report:\n",
    "\n",
    "    <img src= \"inserted-finding.png\" alt=\"Screenshot showing the validation report with the new finding linked\" style=\"border: 2px solid #083E44; border-radius: 8px; border-right-width: 2px; border-bottom-width: 3px;\">\n",
    "    <br><br>\n",
    "\n",
    "11. Click on the finding to expand the finding, where you can adjust details such as severity, owner, due date, status, etc. as well as include proposed remediation plans or supporting documentation as attachments."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc6_1_3_'></a>\n",
    "\n",
    "#### Evaluate performance of challenger model\n",
    "\n",
    "We've now conducted similar tests as the model development team for our champion model, with the aim of verifying their test results.\n",
    "\n",
    "Next, let's see how our challenger models compare. We'll use the same batch of tests here as we did in `mpt`, but append a different `result_id` to indicate that these results should be associated with our challenger model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mpt_chall = [\n",
    "    \"validmind.model_validation.sklearn.ClassifierPerformance:champion_vs_challenger\",\n",
    "    \"validmind.model_validation.sklearn.ConfusionMatrix:champion_vs_challenger\",\n",
    "    \"validmind.model_validation.sklearn.MinimumAccuracy:champion_vs_challenger\",\n",
    "    \"validmind.model_validation.sklearn.MinimumF1Score:champion_vs_challenger\",\n",
    "    \"validmind.model_validation.sklearn.ROCCurve:champion_vs_challenger\"\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll run each test once for each model with the same `vm_test_ds` dataset to compare them:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for test in mpt_chall:\n",
    "    vm.tests.run_test(\n",
    "        test,\n",
    "        input_grid={\n",
    "            \"dataset\": [vm_test_ds], \"model\" : [vm_log_model,vm_rf_model]\n",
    "        }\n",
    "    ).log()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\" style=\"background-color: #B5B5B510; color: black; border: 1px solid #083E44; border-left-width: 5px; box-shadow: 2px 2px 4px rgba(0, 0, 0, 0.2);border-radius: 5px;\"><span style=\"color: #083E44;\"><b>Based on the performance metrics, our challenger random forest classification model passes the <code>MinimumAccuracy</code> where our champion did not.</b></span>\n",
    "<br></br>\n",
    "In your validation report, support your recommendation in your finding's <b>Proposed Remediation Plan</b> to investigate the usage of our challenger model by inserting the performance tests we logged with this notebook into the appropriate section.</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc6_2_'></a>\n",
    "\n",
    "### Run diagnostic tests\n",
    "\n",
    "Next, we want to inspect the robustness and stability testing comparison between our champion and challenger model.\n",
    "\n",
    "Use `list_tests()` to list all available diagnosis tests applicable to classification tasks:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vm.tests.list_tests(tags=[\"model_diagnosis\"], task=\"classification\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let’s now assess the models for potential signs of *overfitting* and identify any sub-segments where performance may inconsistent with the [`OverfitDiagnosis` test](https://docs.validmind.ai/tests/model_validation/sklearn/OverfitDiagnosis.html).\n",
    "\n",
    "Overfitting occurs when a model learns the training data too well, capturing not only the true pattern but noise and random fluctuations resulting in excellent performance on the training dataset but poor generalization to new, unseen data:\n",
    "\n",
    "- Since the training dataset (`vm_train_ds`) was used to fit the model, we use this set to establish a baseline performance for how well the model performs on data it has already seen.\n",
    "- The testing dataset (`vm_test_ds`) was never seen during training, and here simulates real-world generalization, or how well the model performs on new, unseen data. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vm.tests.run_test(\n",
    "    test_id=\"validmind.model_validation.sklearn.OverfitDiagnosis:champion_vs_challenger\",\n",
    "    input_grid={\n",
    "        \"datasets\": [[vm_train_ds,vm_test_ds]],\n",
    "        \"model\" : [vm_log_model,vm_rf_model]\n",
    "    }\n",
    ").log()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's also conduct *robustness* and *stability* testing of the two models with the [`RobustnessDiagnosis` test](https://docs.validmind.ai/tests/model_validation/sklearn/RobustnessDiagnosis.html). Robustness refers to a model's ability to maintain consistent performance, and stability refers to a model's ability to produce consistent outputs over time across different data subsets.\n",
    "\n",
    "Again, we'll use both the training and testing datasets to establish baseline performance and to simulate real-world generalization:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vm.tests.run_test(\n",
    "    test_id=\"validmind.model_validation.sklearn.RobustnessDiagnosis:Champion_vs_LogRegression\",\n",
    "    input_grid={\n",
    "        \"datasets\": [[vm_train_ds,vm_test_ds]],\n",
    "        \"model\" : [vm_log_model,vm_rf_model]\n",
    "    },\n",
    ").log()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc6_3_'></a>\n",
    "\n",
    "### Run feature importance tests\n",
    "\n",
    "We also want to verify the relative influence of different input features on our models' predictions, as well as inspect the differences between our champion and challenger model to see if a certain model offers more understandable or logical importance scores for features.\n",
    "\n",
    "Use `list_tests()` to identify all the feature importance tests for classification:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Store the feature importance tests\n",
    "FI = vm.tests.list_tests(tags=[\"feature_importance\"], task=\"classification\",pretty=False)\n",
    "FI"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll only use our testing dataset (`vm_test_ds`) here, to provide a realistic, unseen sample that mimic future or production data, as the training dataset has already influenced our model during learning:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run and log our feature importance tests for both models for the testing dataset\n",
    "for test in FI:\n",
    "    vm.tests.run_test(\n",
    "        \"\".join((test,':champion_vs_challenger')),\n",
    "        input_grid={\n",
    "            \"dataset\": [vm_test_ds], \"model\" : [vm_log_model,vm_rf_model]\n",
    "        },\n",
    "    ).log()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc7_'></a>\n",
    "\n",
    "## In summary\n",
    "\n",
    "In this third notebook, you learned how to:\n",
    "\n",
    "- [x] Initialize ValidMind model objects\n",
    "- [x] Assign predictions and probabilities to your ValidMind model objects\n",
    "- [x] Use tests from ValidMind to evaluate the potential of models, including comparative tests between champion and challenger models\n",
    "- [x] Log a model finding in the ValidMind Platform"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc8_'></a>\n",
    "\n",
    "## Next steps\n",
    "\n",
    "<a id='toc8_1_'></a>\n",
    "\n",
    "### Finalize validation and reporting\n",
    "\n",
    "Now that you're familiar with the basics of using the ValidMind Library to run and log validation tests, let's learn how to implement some custom tests and wrap up our validation: **[4 — Finalize validation and reporting](4-finalize_validation_reporting.ipynb)**"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "ValidMind Library",
   "language": "python",
   "name": "validmind"
  },
  "language_info": {
   "name": "python",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
