{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction to ValidMind Dataset and Model Objects\n",
    "\n",
    "When writing custom tests, it is essential to be aware of the interfaces of the ValidMind Dataset and ValidMind Model, which are used as input arguments.\n",
    "\n",
    "As a model developer, writing custom tests is beneficial when the ValidMind library lacks a built-in test for your specific needs. For example, a model might require new tests to evaluate specific aspects of the model or dataset based on a particular use case.\n",
    "\n",
    "This interactive notebook offers a detailed understanding of ValidMind objects and their use in writing custom tests. It introduces various interfaces provided by these objects and demonstrates how they can be leveraged to implement tests effortlessly."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Contents\n",
    "- [About ValidMind](#toc1_)    \n",
    "  - [Before you begin](#toc1_1_)    \n",
    "  - [New to ValidMind?](#toc1_2_)    \n",
    "  - [Key concepts](#toc1_3_)    \n",
    "- [Install the ValidMind Library](#toc2_)    \n",
    "- [Initialize the ValidMind Library](#toc3_)   \n",
    "  - [Get your code snippet](#toc3_1_)    \n",
    "  <!-- - [Preview the documentation template](#toc3_1_)     -->\n",
    "- [Load the demo dataset](#toc4_)    \n",
    "  - [Prepocess the raw dataset](#toc4_1_)    \n",
    "- [Train a model for testing](#toc5_)    \n",
    "- [Explore basic components of the ValidMind library](#toc6_)    \n",
    "  - [VMDataset Object](#toc6_1_)\n",
    "    - [Initialize the ValidMind datasets](#toc6_1_1_)\n",
    "    - [Interfaces of the dataset object](#toc6_1_2_)\n",
    "    - [Using VM Dataset object as arguments in Custom tests](#toc6_1_3_)\n",
    "    - [Using VM Dataset object and parameters as arguments in Custom tests](#toc6_1_4_)  \n",
    "  - [VMModel Object](#toc6_2_)\n",
    "    - [Initialize ValidMind model object](#toc6_2_1_)\n",
    "    - [Assign predictions to the datasets](#toc6_2_2_)    \n",
    "    - [Using VM Model and Dataset objects as arguments in Custom tests](#toc6_2_3_)\n",
    "  - [Log the test results](#toc6_4_)\n",
    "- [Where to go from here](#toc7_)\n",
    "  - [Use cases](#toc7_1_)\n",
    "  - [More how-to guides and code samples](#toc7_2_)\n",
    "  - [Discover more learning resources](#toc7_3_)\n",
    "- [Upgrade ValidMind](#toc8_)\n",
    "\n",
    "\n",
    "<!-- vscode-jupyter-toc-config\n",
    "\tnumbering=false\n",
    "\tanchor=true\n",
    "\tflat=false\n",
    "\tminLevel=2\n",
    "\tmaxLevel=4\n",
    "\t/vscode-jupyter-toc-config -->\n",
    "<!-- THIS CELL WILL BE REPLACED ON TOC UPDATE. DO NOT WRITE YOUR TEXT IN THIS CELL -->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc1_'></a>\n",
    "\n",
    "## About ValidMind\n",
    "\n",
    "ValidMind is a suite of tools for managing model risk, including risk associated with AI and statistical models. You use the ValidMind Library to automate documentation and validation tests, and then use the ValidMind Platform to collaborate on model documentation. Together, these products simplify model risk management, facilitate compliance with regulations and institutional standards, and enhance collaboration between yourself and model validators.\n",
    "\n",
    "<a id='toc1_1_'></a>\n",
    "\n",
    "### Before you begin\n",
    "\n",
    "This notebook assumes you have basic familiarity with Python, including an understanding of how functions work. If you are new to Python, you can still run the notebook but we recommend further familiarizing yourself with the language.\n",
    "\n",
    "If you encounter errors due to missing modules in your Python environment, install the modules with `pip install`, and then re-run the notebook. For more help, refer to [Installing Python Modules](https://docs.python.org/3/installing/index.html).\n",
    "\n",
    "<a id='toc1_2_'></a>\n",
    "\n",
    "### New to ValidMind?\n",
    "\n",
    "If you haven't already seen our documentation on the [ValidMind Library](https://docs.validmind.ai/developer/validmind-library.html), we recommend you begin by exploring the available resources in this section. There, you can learn more about documenting models and running tests, as well as find code samples and our Python Library API reference.\n",
    "\n",
    "<div class=\"alert alert-block alert-info\" style=\"background-color: #B5B5B510; color: black; border: 1px solid #083E44; border-left-width: 5px; box-shadow: 2px 2px 4px rgba(0, 0, 0, 0.2);border-radius: 5px;\"><span style=\"color: #083E44;\"><b>For access to all features available in this notebook, create a free ValidMind account.</b></span>\n",
    "<br></br>\n",
    "Signing up is FREE — <a href=\"https://docs.validmind.ai/guide/configuration/register-with-validmind.html\" style=\"color: #DE257E;\"><b>Register with ValidMind</b></a></div>\n",
    "\n",
    "<a id='toc1_3_'></a>\n",
    "\n",
    "### Key concepts\n",
    "\n",
    "Here, we will focus on ValidMind dataset, ValidMind model and tests to use these objects to generate artefacts for the documentation.\n",
    "\n",
    "**Tests**: A function contained in the ValidMind Library, designed to run a specific quantitative test on the dataset or model. Tests are the building blocks of ValidMind, used to evaluate and document models and datasets, and can be run individually or as part of a suite defined by your model documentation template.\n",
    "\n",
    "**Custom tests**: Custom tests are functions that you define to evaluate your model or dataset. These functions can be registered via the ValidMind Library to be used with the ValidMind Platform.\n",
    "\n",
    "**Inputs**: Objects to be evaluated and documented in the ValidMind Library. They can be any of the following:\n",
    "\n",
    "- **model**: A single ValidMind model object that has been initialized in ValidMind with [`vm.init_model()`](https://docs.validmind.ai/validmind/validmind.html#init_model).\n",
    "- **dataset**: Single Validmind dataset object that has been initialized in ValidMind with [`vm.init_dataset()`](https://docs.validmind.ai/validmind/validmind.html#init_dataset).\n",
    "- **models**: A list of ValidMind models - usually this is used when you want to compare multiple models in your custom test.\n",
    "- **datasets**: A list of ValidMind datasets - usually this is used when you want to compare multiple datasets in your custom test. See this [example](https://docs.validmind.ai/notebooks/how_to/run_tests_that_require_multiple_datasets.html) for more information.\n",
    "\n",
    "**Parameters**: Additional arguments that can be passed when running a ValidMind test, used to pass additional information to a test, customize its behavior, or provide additional context.\n",
    "\n",
    "**Outputs**: Tests can return elements like tables or plots. Tables may be a list of dictionaries (each representing a row) or a pandas DataFrame. Plots may be matplotlib or plotly figures.\n",
    "\n",
    "**Dataset based Test**\n",
    "\n",
    "![Dataset based test architecture](./dataset_image.png)\n",
    "The dataset based tests take VM dataset object(s) as inputs, test configuration as test parameters to produce `Outputs` as mentioned above.\n",
    "\n",
    "**Model based Test**\n",
    "\n",
    "![Model based test architecture](./model_image.png)\n",
    "Similar to datasest based tests, the model based tests as an additional input that is VM model object. It allows to identify prediction values of a specific model in the dataset object. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc2_'></a>\n",
    "\n",
    "## Install the ValidMind Library\n",
    "\n",
    "Please note the following recommended Python versions to use:\n",
    "\n",
    "- Python 3.7 > x <= 3.11\n",
    "\n",
    "To install the library:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "metadata": {}
   },
   "outputs": [],
   "source": [
    "%pip install -q validmind"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc3_'></a>\n",
    "\n",
    "## Initialize the ValidMind Library\n",
    "\n",
    "ValidMind generates a unique _code snippet_ for each registered model to connect with your developer environment. You initialize the ValidMind Library with this code snippet, which ensures that your documentation and tests are uploaded to the correct model when you run the notebook.\n",
    "\n",
    "<a id='toc3_1_'></a>\n",
    "\n",
    "### Get your code snippet\n",
    "\n",
    "1. In a browser, [log in to ValidMind](https://docs.validmind.ai/guide/configuration/log-in-to-validmind.html).\n",
    "\n",
    "2. In the left sidebar, navigate to **Model Inventory** and click **+ Register Model**.\n",
    "\n",
    "3. Enter the model details and click **Continue**. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/register-models-in-inventory.html))\n",
    "\n",
    "   For example, to register a model for use with this notebook, select:\n",
    "\n",
    "   - Documentation template: `Binary classification`\n",
    "   - Use case: `Marketing/Sales - Attrition/Churn Management`\n",
    "\n",
    "   You can fill in other options according to your preference.\n",
    "\n",
    "4. Go to **Getting Started** and click **Copy snippet to clipboard**.\n",
    "\n",
    "Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "metadata": {}
   },
   "outputs": [],
   "source": [
    "# Load your model identifier credentials from an `.env` file\n",
    "\n",
    "%load_ext dotenv\n",
    "%dotenv .env\n",
    "\n",
    "# Or replace with your code snippet\n",
    "\n",
    "import validmind as vm\n",
    "\n",
    "vm.init(\n",
    "    # api_host=\"...\",\n",
    "    # api_key=\"...\",\n",
    "    # api_secret=\"...\",\n",
    "    # model=\"...\",\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "import xgboost as xgb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc4_'></a>\n",
    "\n",
    "## Load the demo dataset\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from validmind.datasets.classification import customer_churn as demo_dataset\n",
    "\n",
    "raw_df = demo_dataset.load_data()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc4_1_'></a>\n",
    "\n",
    "### Prepocess the raw dataset\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_df, validation_df, test_df = demo_dataset.preprocess(raw_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc5_'></a>\n",
    "\n",
    "## Train a model for testing\n",
    "\n",
    "We train a simple customer churn model for our test.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_train = train_df.drop(demo_dataset.target_column, axis=1)\n",
    "y_train = train_df[demo_dataset.target_column]\n",
    "x_val = validation_df.drop(demo_dataset.target_column, axis=1)\n",
    "y_val = validation_df[demo_dataset.target_column]\n",
    "\n",
    "model = xgb.XGBClassifier(early_stopping_rounds=10)\n",
    "model.set_params(\n",
    "    eval_metric=[\"error\", \"logloss\", \"auc\"],\n",
    ")\n",
    "model.fit(\n",
    "    x_train,\n",
    "    y_train,\n",
    "    eval_set=[(x_val, y_val)],\n",
    "    verbose=False,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc6_'></a>\n",
    "\n",
    "## Explore basic components of the ValidMind library\n",
    "\n",
    "In this section, you will learn about the basic objects of the ValidMind library that are necessary to implement both custom and built-in tests. As explained above, these objects are:\n",
    "* VMDataset: [The high level APIs can be found here](https://docs.validmind.ai/validmind/validmind/vm_models.html#VMDataset)\n",
    "* VMModel: [The high level APIs can be found here](https://docs.validmind.ai/validmind/validmind/vm_models.html#VMModel)\n",
    "\n",
    "Let's understand these objects and their interfaces step by step: "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc6_1_'></a>\n",
    "\n",
    "### VMDataset Object"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc6_1_1_'></a>\n",
    "\n",
    "#### Initialize the ValidMind datasets\n",
    "\n",
    "You can initialize a ValidMind dataset object using the [`init_dataset`](https://docs.validmind.ai/validmind/validmind.html#init_dataset) function from the ValidMind (`vm`) module.\n",
    "\n",
    "The function wraps the dataset to create a ValidMind `Dataset` object so that you can write tests effectively using the common interface provided by the VM objects. This step is always necessary every time you want to connect a dataset to documentation and produce test results through ValidMind. You only need to do it one time per dataset.\n",
    "\n",
    "This function takes a number of arguments. Some of the arguments are:\n",
    "\n",
    "- `dataset` — the raw dataset that you want to provide as input to tests\n",
    "- `input_id` - a unique identifier that allows tracking what inputs are used when running each individual test\n",
    "- `target_column` — a required argument if tests require access to true values. This is the name of the target column in the dataset\n",
    "\n",
    "The detailed list of the arguments can be found [here](https://docs.validmind.ai/validmind/validmind.html#init_dataset) \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "metadata": {}
   },
   "outputs": [],
   "source": [
    "# vm_raw_dataset is now a VMDataset object that you can pass to any ValidMind test\n",
    "vm_raw_dataset = vm.init_dataset(\n",
    "    dataset=raw_df,\n",
    "    input_id=\"raw_dataset\",\n",
    "    target_column=\"Exited\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once you have a ValidMind dataset object (VMDataset), you can inspect its attributes and methods using the inspect_obj utility module. This module provides a list of available attributes and interfaces for use in tests. Understanding how to use VMDatasets is crucial for comprehending how a custom test functions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from validmind.utils import inspect_obj\n",
    "inspect_obj(vm_raw_dataset)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc6_1_2_'></a>\n",
    "\n",
    "####  Interfaces of the dataset object\n",
    "\n",
    "**DataFrame**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vm_raw_dataset.df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Feature columns**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vm_raw_dataset.feature_columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Target column**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vm_raw_dataset.target_column"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Features values**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vm_raw_dataset.x_df()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Target value**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vm_raw_dataset.y_df()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Numeric feature columns** "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vm_raw_dataset.feature_columns_numeric"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Categorical feature columns** "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vm_raw_dataset.feature_columns_categorical"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similarly, you can use all other interfaces of the [VMDataset objects](https://docs.validmind.ai/validmind/validmind/vm_models.html#VMDataset) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc6_1_3_'></a>\n",
    "\n",
    "### Using VM Dataset object as arguments in custom tests\n",
    "\n",
    "A custom test is simply a Python function that takes two types of arguments: `inputs` and `params`. The `inputs` are ValidMind objects (`VMDataset`, `VMModel`), and the `params` are additional parameters required for the underlying computation of the test. We will discuss both types of arguments in the following sections.\n",
    "\n",
    "Let's start with a custom test that requires only a ValidMind dataset object. In this example, we will check the balance of classes in the target column of the dataset:\n",
    "\n",
    "- The custom test below requires a single argument of type `VMDataset` (dataset).\n",
    "- The `my_custom_tests.ClassImbalance` is a unique test identifier that can be assigned using the `vm.test` decorator functionality. This unique test ID will be used in the platform to load test results in the documentation.\n",
    "- The `dataset.target_column` and `dataset.df` attributes of the `VMDataset` object are used in the test.\n",
    "\n",
    "Other high-level APIs (attributes and methods) of the dataset object are listed [here](https://docs.validmind.ai/validmind/validmind/vm_models.html#VMDataset).\n",
    "\n",
    "If you've gone through the [Implement custom tests notebook](../code_samples/custom_tests/implement_custom_tests.ipynb), you should have a good understanding of how custom tests are implemented in details. If you haven't, we recommend going through that notebook first."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from validmind.vm_models.dataset.dataset import VMDataset\n",
    "import pandas as pd\n",
    "\n",
    "@vm.test(\"my_custom_tests.ClassImbalance\")\n",
    "def class_imbalance(dataset):\n",
    "    # Can only run this test if we have a Dataset object\n",
    "    if not isinstance(dataset, VMDataset):\n",
    "        raise ValueError(\"ClassImbalance requires a validmind Dataset object\")\n",
    "\n",
    "    if dataset.target_column is None:\n",
    "        print(\"Skipping class_imbalance test because no target column is defined\")\n",
    "        return\n",
    "\n",
    "    # VMDataset object provides target_column attribute\n",
    "    target_column = dataset.target_column\n",
    "    # we can access pandas DataFrame using df attribute\n",
    "    imbalance_percentages = dataset.df[target_column].value_counts(\n",
    "        normalize=True\n",
    "    )\n",
    "    classes = list(imbalance_percentages.index) \n",
    "    percentages = list(imbalance_percentages.values * 100)\n",
    "\n",
    "    return pd.DataFrame({\"Classes\":classes, \"Percentage\": percentages})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Run the test\n",
    "\n",
    "Let's run the test using the `run_test` method, which is part of the `validmind.tests` module. Here, we pass the `dataset` through the `inputs`. Similarly, you can pass `datasets`, `model`, or `models` as inputs if your custom test requires them. In this example below, we run the custom test `my_custom_tests.ClassImbalance` by passing the `dataset` through the `inputs`. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from validmind.tests import run_test\n",
    "result = run_test(\n",
    "    test_id=\"my_custom_tests.ClassImbalance\",\n",
    "    inputs={\n",
    "        \"dataset\": vm_raw_dataset\n",
    "    }\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can move custom tests into separate modules in a folder. It allows you to take one-off tests and move them into an organized structure that makes it easier to manage, maintain and share them. We have provided a seperate notebook with detailed explaination [here](../code_samples/custom_tests/integrate_external_test_providers.ipynb) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc6_1_4_'></a>\n",
    "\n",
    "### Using VM Dataset object and parameters as arguments in custom tests\n",
    "\n",
    "Simlilar to `inputs`, you can pass `params` to a custom test by providing a dictionary of parameters to the `run_test()` function. The parameters will override any default parameters set in the custom test definition. Note that the `dataset` is still passed as `inputs`. \n",
    "Let's modify the class imbalance test so that it provides flexibility to `normalize` the results.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from validmind.vm_models.dataset.dataset import VMDataset\n",
    "import pandas as pd\n",
    "\n",
    "@vm.test(\"my_custom_tests.ClassImbalance\")\n",
    "def class_imbalance(dataset, normalize=True):\n",
    "    # Can only run this test if we have a Dataset object\n",
    "    if not isinstance(dataset, VMDataset):\n",
    "        raise ValueError(\"ClassImbalance requires a validmind Dataset object\")\n",
    "\n",
    "    if dataset.target_column is None:\n",
    "        print(\"Skipping class_imbalance test because no target column is defined\")\n",
    "        return\n",
    "\n",
    "    # VMDataset object provides target_column attribute\n",
    "    target_column = dataset.target_column\n",
    "    # we can access pandas DataFrame using df attribute\n",
    "    imbalance_percentages = dataset.df[target_column].value_counts(\n",
    "        normalize=normalize\n",
    "    )\n",
    "    classes = list(imbalance_percentages.index) \n",
    "    if normalize:  \n",
    "        result = pd.DataFrame({\"Classes\":classes, \"Percentage\": list(imbalance_percentages.values*100)})\n",
    "    else:\n",
    "        result = pd.DataFrame({\"Classes\":classes, \"Count\": list(imbalance_percentages.values)})\n",
    "    return result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this example, the `normalize` parameter is set to `False`, so the class counts will not be normalized. You can change the value to `True` if you want the counts to be normalized. The results of the test will reflect this flexibility, allowing for different outputs based on the parameter passed.\n",
    "\n",
    "Here, we have passed the `dataset` through the `inputs` and the `normalize` parameter using the `params`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from validmind.tests import run_test\n",
    "result = run_test(\n",
    "    test_id = \"my_custom_tests.ClassImbalance\",\n",
    "    inputs={\"dataset\": vm_raw_dataset},\n",
    "    params={\"normalize\": True},\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc6_2_'></a>\n",
    "\n",
    "### VMModel Object\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc6_2_1_'></a>\n",
    "\n",
    "### Initialize ValidMind model object\n",
    "\n",
    "Similar to ValidMind `Dataset` object, you can initialize a ValidMind Model object using the [`init_model`](https://docs.validmind.ai/validmind/validmind.html#init_model) function from the ValidMind (`vm`) module.\n",
    "\n",
    "This function takes a number of arguments. Some of the arguments are:\n",
    "\n",
    "- `model` — the raw model that you want evaluate\n",
    "- `input_id` - a unique identifier that allows tracking what inputs are used when running each individual test\n",
    "\n",
    "The detailed list of the arguments can be found [here](https://docs.validmind.ai/validmind/validmind.html#init_model) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "vm_model = vm.init_model(\n",
    "    model=model,\n",
    "    input_id=\"xgb_model\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's inspect the methods and attributes of the model now:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "inspect_obj(vm_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc6_2_2_'></a>\n",
    "\n",
    "### Assign predictions to the datasets\n",
    "\n",
    "We can now use the `assign_predictions()` method from the `Dataset` object to link existing predictions to any model. If no prediction values are passed, the method will compute predictions automatically:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vm_train_ds = vm.init_dataset(\n",
    "    input_id=\"train_dataset\",\n",
    "    dataset=train_df,\n",
    "    type=\"generic\",\n",
    "    target_column=demo_dataset.target_column,\n",
    ")\n",
    "\n",
    "vm_train_ds.assign_predictions(model=vm_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can see below, the extra prediction column (`xgb_model_prediction`) for the model (`xgb_model`) has been added in the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(vm_train_ds)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc6_2_3_'></a>\n",
    "### Using VM Model and Dataset objects as arguments in Custom tests\n",
    "\n",
    "We will now create a `@vm.test` wrapper that will allow you to create a reusable test. Note the following changes in the code below:\n",
    "\n",
    "- The function `confusion_matrix` takes two arguments `dataset` and `model`. This is a `VMDataset` and `VMModel` object respectively.\n",
    "  - `VMDataset` objects allow you to access the dataset's true (target) values by accessing the `.y` attribute.\n",
    "  - `VMDataset` objects allow you to access the predictions for a given model by accessing the `.y_pred()` method.\n",
    "- The function docstring provides a description of what the test does. This will be displayed along with the result in this notebook as well as in the ValidMind Platform.\n",
    "- The function body calculates the confusion matrix using the `sklearn.tests.confusion_matrix` function as we just did above.\n",
    "- The function then returns the `ConfusionMatrixDisplay.figure_` object - this is important as the ValidMind Library expects the output of the custom test to be a plot or a table.\n",
    "- The `@vm.test` decorator is doing the work of creating a wrapper around the function that will allow it to be run by the ValidMind Library. It also registers the test so it can be found by the ID `my_custom_tests.ConfusionMatrix` (see the section below on how test IDs work in ValidMind and why this format is important)\n",
    "\n",
    "Similarly, you can use the functinality provided by `VMDataset` and `VMModel` objects. You can refer our documentation page for all the avalialble APIs [here](https://docs.validmind.ai/validmind/validmind.html#init_dataset)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn import metrics\n",
    "import matplotlib.pyplot as plt\n",
    "@vm.test(\"my_custom_tests.ConfusionMatrix\")\n",
    "def confusion_matrix(dataset, model):\n",
    "    \"\"\"The confusion matrix is a table that is often used to describe the performance of a classification model on a set of data for which the true values are known.\n",
    "\n",
    "    The confusion matrix is a 2x2 table that contains 4 values:\n",
    "\n",
    "    - True Positive (TP): the number of correct positive predictions\n",
    "    - True Negative (TN): the number of correct negative predictions\n",
    "    - False Positive (FP): the number of incorrect positive predictions\n",
    "    - False Negative (FN): the number of incorrect negative predictions\n",
    "\n",
    "    The confusion matrix can be used to assess the holistic performance of a classification model by showing the accuracy, precision, recall, and F1 score of the model on a single figure.\n",
    "    \"\"\"\n",
    "    # we can retrieve traget value from dataset which is y attribute\n",
    "    y_true = dataset.y\n",
    "    # The prediction value of a specific model using y_pred method \n",
    "    y_pred = dataset.y_pred(model=model)\n",
    "\n",
    "    confusion_matrix = metrics.confusion_matrix(y_true, y_pred)\n",
    "\n",
    "    cm_display = metrics.ConfusionMatrixDisplay(\n",
    "        confusion_matrix=confusion_matrix, display_labels=[False, True]\n",
    "    )\n",
    "    cm_display.plot()\n",
    "    plt.close()\n",
    "\n",
    "    return cm_display.figure_  # return the figure object itself"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, we run test using two inputs; `dataset` and `model`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from validmind.tests import run_test\n",
    "result = run_test(\n",
    "    test_id = \"my_custom_tests.ConfusionMatrix\",\n",
    "    inputs={\n",
    "        \"dataset\": vm_train_ds,\n",
    "        \"model\": vm_model,\n",
    "    }\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc6_4_'></a>\n",
    "\n",
    "### Log the test results\n",
    "\n",
    "You can log any test result to the ValidMind Platform with the `.log()` method of the result object. This will allow you to add the result to the documentation.\n",
    "\n",
    "You can now do the same for the confusion matrix results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "metadata": {}
   },
   "outputs": [],
   "source": [
    "result.log()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc7_'></a>\n",
    "\n",
    "## Where to go from here\n",
    "\n",
    "In this notebook you have learned the end-to-end process to document a model with the ValidMind Library, running through some very common scenarios in a typical model development setting:\n",
    "\n",
    "- Running out-of-the-box tests\n",
    "- Documenting your model by adding evidence to model documentation\n",
    "- Extending the capabilities of the ValidMind Library by implementing custom tests\n",
    "- Ensuring that the documentation is complete by running all tests in the documentation template\n",
    "\n",
    "As a next step, you can explore the following notebooks to get a deeper understanding on how the ValidMind Library allows you generate model documentation for any use case:\n",
    "\n",
    "<a id='toc7_1_'></a>\n",
    "\n",
    "### Use cases\n",
    "\n",
    "- [Application scorecard demo](../code_samples/credit_risk/application_scorecard_demo.ipynb)\n",
    "- [Linear regression documentation demo](../code_samples/regression/quickstart_regression_full_suite.ipynb)\n",
    "- [LLM model documentation demo](../code_samples/nlp_and_llm/foundation_models_integration_demo.ipynb)\n",
    "\n",
    "<a id='toc7_2_'></a>\n",
    "\n",
    "### More how-to guides and code samples\n",
    "\n",
    "- [Explore available tests in detail](../how_to/explore_tests.ipynb)\n",
    "- [In-depth guide for implementing custom tests](../code_samples/custom_tests/implement_custom_tests.ipynb)\n",
    "- [In-depth guide to external test providers](../code_samples/custom_tests/integrate_external_test_providers.ipynb)\n",
    "- [Configuring dataset features](../how_to/configure_dataset_features.ipynb)\n",
    "- [Introduction to unit and composite tests](../how_to/run_unit_metrics.ipynb)\n",
    "\n",
    "<a id='toc7_3_'></a>\n",
    "\n",
    "### Discover more learning resources\n",
    "\n",
    "All notebook samples can be found in the following directories of the ValidMind Library GitHub repository:\n",
    "\n",
    "- [Code samples](https://github.com/validmind/validmind-library/tree/main/notebooks/code_samples)\n",
    "- [How-to guides](https://github.com/validmind/validmind-library/tree/main/notebooks/how_to)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc8_'></a>\n",
    "\n",
    "## Upgrade ValidMind\n",
    "\n",
    "<div class=\"alert alert-block alert-info\" style=\"background-color: #B5B5B510; color: black; border: 1px solid #083E44; border-left-width: 5px; box-shadow: 2px 2px 4px rgba(0, 0, 0, 0.2);border-radius: 5px;\">After installing ValidMind, you’ll want to periodically make sure you are on the latest version to access any new features and other enhancements.</div>\n",
    "\n",
    "Retrieve the information for the currently installed version of ValidMind:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip show validmind"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If the version returned is lower than the version indicated in our [production open-source code](https://github.com/validmind/validmind-library/blob/prod/validmind/__version__.py), restart your notebook and run:\n",
    "\n",
    "```bash\n",
    "%pip install --upgrade validmind\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You may need to restart your kernel after running the upgrade package for changes to be applied."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
