{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Add context to LLM-generated test descriptions\n",
    "\n",
    "When you run ValidMind tests, test descriptions are automatically generated with LLM using the test results, the test name, and the static test definitions provided in the test's docstring. While this metadata offers valuable high-level overviews of tests, insights produced by the LLM-based descriptions may not always align with your specific use cases or incorporate organizational policy requirements.\n",
    "\n",
    "In this notebook, you'll learn how to add context to the generated descriptions by providing additional information about the test or the use case. Including custom use case context is useful when you want to highlight information about the intended use and technique of the model, or the insitution policies and standards specific to your use case."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "::: {.content-hidden when-format=\"html\"}\n",
    "## Contents    \n",
    "- [Install the ValidMind Library](#toc1_)    \n",
    "- [Initialize the ValidMind Library](#toc2_)    \n",
    "  - [Get your code snippet](#toc2_1_)    \n",
    "- [Initialize the Python environment](#toc3_)    \n",
    "- [Load the sample dataset](#toc4_)    \n",
    "  - [Preprocess the raw dataset](#toc4_1_)    \n",
    "- [Initializing the ValidMind objects](#toc5_)    \n",
    "  - [Initialize the datasets](#toc5_1_)    \n",
    "  - [Initialize a model object](#toc5_2_)    \n",
    "  - [Assign predictions to the datasets](#toc5_3_)    \n",
    "- [Set custom context for test descriptions](#toc6_)    \n",
    "  - [Review default LLM-generated descriptions](#toc6_1_)    \n",
    "  - [Enable use case context](#toc6_2_)    \n",
    "    - [Disable use case context](#toc6_2_1_)    \n",
    "  - [Add test-specific context](#toc6_3_)    \n",
    "    - [Dataset Description](#toc6_3_1_)    \n",
    "    - [Class Imbalance](#toc6_3_2_)    \n",
    "    - [High Cardinality](#toc6_3_3_)    \n",
    "    - [Missing Values](#toc6_3_4_)    \n",
    "    - [Unique Rows](#toc6_3_5_)    \n",
    "    - [Too Many Zero Values](#toc6_3_6_)    \n",
    "    - [IQR Outliers Table](#toc6_3_7_)    \n",
    "    - [Descriptive Statistics](#toc6_3_8_)    \n",
    "    - [Pearson Correlation Matrix](#toc6_3_9_)    \n",
    "  - [Add test-specific context using the docstring](#toc6_4_)\n",
    "- [Best practices for adding custom context](#toc7_)\n",
    ":::\n",
    "<!-- jn-toc-notebook-config\n",
    "\tnumbering=false\n",
    "\tanchor=true\n",
    "\tflat=false\n",
    "\tminLevel=2\n",
    "\tmaxLevel=4\n",
    "\t/jn-toc-notebook-config -->\n",
    "<!-- THIS CELL WILL BE REPLACED ON TOC UPDATE. DO NOT WRITE YOUR TEXT IN THIS CELL -->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc1_'></a>\n",
    "\n",
    "## Install the ValidMind Library\n",
    "\n",
    "To install the library:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install -q validmind"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc2_'></a>\n",
    "\n",
    "## Initialize the ValidMind Library\n",
    "\n",
    "ValidMind generates a unique _code snippet_ for each registered model to connect with your developer environment. You initialize the ValidMind Library with this code snippet, which ensures that your documentation and tests are uploaded to the correct model when you run the notebook.\n",
    "\n",
    "<a id='toc3_1_'></a>\n",
    "\n",
    "<a id='toc2_1_'></a>\n",
    "\n",
    "### Get your code snippet\n",
    "\n",
    "1. In a browser, [log in to ValidMind](https://docs.validmind.ai/guide/configuration/log-in-to-validmind.html).\n",
    "\n",
    "2. In the left sidebar, navigate to **Model Inventory** and click **+ Register Model**.\n",
    "\n",
    "3. Enter the model details and click **Continue**. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/register-models-in-inventory.html))\n",
    "\n",
    "   For example, to register a model for use with this notebook, select:\n",
    "\n",
    "   - Documentation template: `Binary classification`\n",
    "   - Use case: `Marketing/Sales - Attrition/Churn Management`\n",
    "\n",
    "   You can fill in other options according to your preference.\n",
    "\n",
    "4. Go to **Getting Started** and click **Copy snippet to clipboard**.\n",
    "\n",
    "Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load your model identifier credentials from an `.env` file\n",
    "\n",
    "%load_ext dotenv\n",
    "%dotenv .env\n",
    "\n",
    "# Or replace with your code snippet\n",
    "\n",
    "import validmind as vm\n",
    "\n",
    "vm.init(\n",
    "    # api_host=\"...\",\n",
    "    # api_key=\"...\",\n",
    "    # api_secret=\"...\",\n",
    "    # model=\"...\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc3_'></a>\n",
    "\n",
    "## Initialize the Python environment\n",
    "\n",
    "After you've connected to your model register in the ValidMind Platform, let's import the necessary libraries and set up your Python environment for data analysis:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import xgboost as xgb\n",
    "import os\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc4_'></a>\n",
    "\n",
    "## Load the sample dataset\n",
    "\n",
    "First, we'll import a sample ValidMind dataset and load it into a pandas [DataFrame](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.html), a two-dimensional tabular data structure that makes use of rows and columns:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import the sample dataset from the library\n",
    "\n",
    "from validmind.datasets.classification import customer_churn\n",
    "\n",
    "print(\n",
    "    f\"Loaded demo dataset with: \\n\\n\\t• Target column: '{customer_churn.target_column}' \\n\\t• Class labels: {customer_churn.class_labels}\"\n",
    ")\n",
    "\n",
    "raw_df = customer_churn.load_data()\n",
    "raw_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc4_1_'></a>\n",
    "\n",
    "### Preprocess the raw dataset\n",
    "\n",
    "Then, we'll perform a number of operations to get ready for the subsequent steps:\n",
    "\n",
    "- **Preprocess the data:** Splits the DataFrame (`df`) into multiple datasets (`train_df`, `validation_df`, and `test_df`) using `demo_dataset.preprocess` to simplify preprocessing.\n",
    "- **Separate features and targets:** Drops the target column to create feature sets (`x_train`, `x_val`) and target sets (`y_train`, `y_val`).\n",
    "- **Initialize XGBoost classifier:** Creates an `XGBClassifier` object with early stopping rounds set to 10.\n",
    "- **Set evaluation metrics:** Specifies metrics for model evaluation as `error`, `logloss`, and `auc`.\n",
    "- **Fit the model:** Trains the model on `x_train` and `y_train` using the validation set `(x_val, y_val)`. Verbose output is disabled."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_df, validation_df, test_df = customer_churn.preprocess(raw_df)\n",
    "\n",
    "x_train = train_df.drop(customer_churn.target_column, axis=1)\n",
    "y_train = train_df[customer_churn.target_column]\n",
    "x_val = validation_df.drop(customer_churn.target_column, axis=1)\n",
    "y_val = validation_df[customer_churn.target_column]\n",
    "\n",
    "model = xgb.XGBClassifier(early_stopping_rounds=10)\n",
    "model.set_params(\n",
    "    eval_metric=[\"error\", \"logloss\", \"auc\"],\n",
    ")\n",
    "model.fit(\n",
    "    x_train,\n",
    "    y_train,\n",
    "    eval_set=[(x_val, y_val)],\n",
    "    verbose=False,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc5_'></a>\n",
    "\n",
    "## Initializing the ValidMind objects"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc5_1_'></a>\n",
    "\n",
    "### Initialize the datasets\n",
    "\n",
    "Before you can run tests, you'll need to initialize a ValidMind dataset object using the [`init_dataset`](https://docs.validmind.ai/validmind/validmind.html#init_dataset) function from the ValidMind (`vm`) module.\n",
    "\n",
    "We'll include the following arguments:\n",
    "\n",
    "- **`dataset`** — the raw dataset that you want to provide as input to tests\n",
    "- **`input_id`** - a unique identifier that allows tracking what inputs are used when running each individual test\n",
    "- **`target_column`** — a required argument if tests require access to true values. This is the name of the target column in the dataset\n",
    "- **`class_labels`** — an optional value to map predicted classes to class labels\n",
    "\n",
    "With all datasets ready, you can now initialize the raw, training, and test datasets (`raw_df`, `train_df` and `test_df`) created earlier into their own dataset objects using [`vm.init_dataset()`](https://docs.validmind.ai/validmind/validmind.html#init_dataset):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "vm_raw_dataset = vm.init_dataset(\n",
    "    dataset=raw_df,\n",
    "    input_id=\"raw_dataset\",\n",
    "    target_column=customer_churn.target_column,\n",
    "    class_labels=customer_churn.class_labels,\n",
    ")\n",
    "\n",
    "vm_train_ds = vm.init_dataset(\n",
    "    dataset=train_df,\n",
    "    input_id=\"train_dataset\",\n",
    "    target_column=customer_churn.target_column,\n",
    ")\n",
    "\n",
    "vm_test_ds = vm.init_dataset(\n",
    "    dataset=test_df, input_id=\"test_dataset\", target_column=customer_churn.target_column\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc5_2_'></a>\n",
    "\n",
    "### Initialize a model object\n",
    "\n",
    "Additionally, you'll need to initialize a ValidMind model object (`vm_model`) that can be passed to other functions for analysis and tests on the data. \n",
    "\n",
    "Simply intialize this model object with [`vm.init_model()`](https://docs.validmind.ai/validmind/validmind.html#init_model):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "vm_model = vm.init_model(\n",
    "    model,\n",
    "    input_id=\"model\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc5_3_'></a>\n",
    "\n",
    "### Assign predictions to the datasets\n",
    "\n",
    "We can now use the `assign_predictions()` method from the Dataset object to link existing predictions to any model.\n",
    "\n",
    "If no prediction values are passed, the method will compute predictions automatically:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vm_train_ds.assign_predictions(\n",
    "    model=vm_model,\n",
    ")\n",
    "\n",
    "vm_test_ds.assign_predictions(\n",
    "    model=vm_model,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc6_'></a>\n",
    "\n",
    "## Set custom context for test descriptions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc6_1_'></a>\n",
    "\n",
    "### Review default LLM-generated descriptions\n",
    "\n",
    "By default, custom context for LLM-generated descriptions is disabled, meaning that the output will not include any additional context.\n",
    "\n",
    "Let's generate an initial test description for the `DatasetDescription` test for comparison with later iterations:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vm.tests.run_test(\n",
    "    \"validmind.data_validation.DatasetDescription\",\n",
    "    inputs={\n",
    "        \"dataset\": vm_raw_dataset,\n",
    "    },\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc6_2_'></a>\n",
    "\n",
    "### Enable use case context\n",
    "\n",
    "To enable custom use case context, set the `VALIDMIND_LLM_DESCRIPTIONS_CONTEXT_ENABLED` environment variable to `1`.\n",
    "\n",
    "This is a global setting that will affect all tests for your linked model for the duration of your ValidMind Library session:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "os.environ[\"VALIDMIND_LLM_DESCRIPTIONS_CONTEXT_ENABLED\"] = \"1\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Enabling use case context allows you to pass in additional context, such as information about your model, relevant regulatory requirements, or model validation targets to the LLM-generated text descriptions within `use_case_context`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "use_case_context = \"\"\"\n",
    "\n",
    "This is a customer churn prediction model for a banking loan application system using XGBoost classifier. \n",
    "\n",
    "Key Model Information:\n",
    "- Use Case: Predict customer churn risk during loan application process\n",
    "- Model Type: Binary classification using XGBoost\n",
    "- Critical Decision Point: Used in loan approval workflow\n",
    "\n",
    "Regulatory Requirements:\n",
    "- Subject to model risk management review and validation\n",
    "- Results require validation review for regulatory compliance\n",
    "- Model decisions directly impact loan approval process\n",
    "- Does this result raise any regulatory concerns?\n",
    "\n",
    "Validation Focus:\n",
    "- Explain strengths and weaknesses of the test and the context of whether the result is acceptable.\n",
    "- What does the result indicate about model reliability?\n",
    "- Is the result within acceptable thresholds for loan decisioning?\n",
    "- What are the implications for customer impact?\n",
    "\n",
    "\"\"\".strip()\n",
    "\n",
    "os.environ[\"VALIDMIND_LLM_DESCRIPTIONS_CONTEXT\"] = use_case_context"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the use case context set, generate an updated test description for the `DatasetDescription` test for comparison with default output:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vm.tests.run_test(\n",
    "    \"validmind.data_validation.DatasetDescription\",\n",
    "    inputs={\n",
    "        \"dataset\": vm_raw_dataset,\n",
    "    },\n",
    ").log()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc6_2_1_'></a>\n",
    "\n",
    "#### Disable use case context\n",
    "\n",
    "To disable custom use case context, set the `VALIDMIND_LLM_DESCRIPTIONS_CONTEXT_ENABLED` environment variable to `0`.\n",
    "\n",
    "This is a global setting that will affect all tests for your linked model for the duration of your ValidMind Library session:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "os.environ[\"VALIDMIND_LLM_DESCRIPTIONS_CONTEXT_ENABLED\"] = \"0\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the use case context disabled again, generate another test description for the `DatasetDescription` test for comparison with previous custom output:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vm.tests.run_test(\n",
    "    \"validmind.data_validation.DatasetDescription\",\n",
    "    inputs={\n",
    "        \"dataset\": vm_raw_dataset,\n",
    "    },\n",
    ").log()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc6_3_'></a>\n",
    "\n",
    "### Add test-specific context\n",
    "\n",
    "In addition to the model-level `use_case_context`, you're able to add test-specific context to your LLM-generated descriptions allowing you to provide test-specific validation criteria about the test that is being run.\n",
    "\n",
    "We'll reenable use case context by setting the `VALIDMIND_LLM_DESCRIPTIONS_CONTEXT_ENABLED` environment variable to `1`, then join the test-specific context to the use case context using the `VALIDMIND_LLM_DESCRIPTIONS_CONTEXT` environment variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "os.environ[\"VALIDMIND_LLM_DESCRIPTIONS_CONTEXT_ENABLED\"] = \"1\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc6_3_1_'></a>\n",
    "\n",
    "#### Dataset Description\n",
    "\n",
    "Rather than relying on generic dataset result descriptions in isolation, we'll use the context to specify precise thresholds for missing values, appropriate data types for banking variables (like `CreditScore` and `Balance`), and valid value ranges based on particular business rules:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_context = \"\"\"\n",
    "\n",
    "Acceptance Criteria:\n",
    "- Missing Values: All critical features must have less than 5% missing values (including CreditScore, Balance, Age)\n",
    "- Data Types: All columns must have appropriate data types (numeric for CreditScore/Balance/Age, categorical for Geography/Gender)\n",
    "- Cardinality: Categorical variables must have fewer than 50 unique values, while continuous variables should show appropriate distinct value counts (e.g., high for EstimatedSalary, exactly 2 for Boolean fields)\n",
    "- Value Ranges: Numeric fields must fall within business-valid ranges (CreditScore: 300-850, Age: ≥18, Balance: ≥0)\n",
    "\"\"\".strip()\n",
    "\n",
    "context = f\"\"\"\n",
    "{use_case_context}\n",
    "\n",
    "{test_context}\n",
    "\"\"\".strip()\n",
    "\n",
    "os.environ[\"VALIDMIND_LLM_DESCRIPTIONS_CONTEXT\"] = context"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the test-specific context set, generate an updated test description for the `DatasetDescription` test again:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vm.tests.run_test(\n",
    "    \"validmind.data_validation.DatasetDescription\",\n",
    "    inputs={\n",
    "        \"dataset\": vm_raw_dataset,\n",
    "    },\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc6_3_2_'></a>\n",
    "\n",
    "#### Class Imbalance\n",
    "\n",
    "The following test-specific context example adds value to the LLM-generated description by providing defined risk levels to assess class representation:\n",
    "\n",
    "- By categorizing classes into `Low`, `Medium`, and `High Risk`, the LLM can generate more nuanced and actionable insights, ensuring that the analysis aligns with business requirements for balanced datasets.\n",
    "- This approach not only highlights potential issues but also guides necessary documentation and mitigation strategies for high-risk classes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_context = \"\"\"\n",
    "\n",
    "Acceptance Criteria:\n",
    "\n",
    "• Risk Levels for Class Representation:\n",
    "  - Low Risk: Each class represents 20% or more of the total dataset\n",
    "  - Medium Risk: Each class represents between 10% and 19.9% of the total dataset\n",
    "  - High Risk: Any class represents less than 10% of the total dataset\n",
    "\n",
    "• Overall Requirement:\n",
    "  - All classes must achieve at least Medium Risk status to pass\n",
    "\"\"\".strip()\n",
    "\n",
    "context = f\"\"\"\n",
    "{use_case_context}\n",
    "\n",
    "{test_context}\n",
    "\"\"\".strip()\n",
    "\n",
    "os.environ[\"VALIDMIND_LLM_DESCRIPTIONS_CONTEXT\"] = context"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the test-specific context set, generate a test description for the `ClassImbalance` test for review:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vm.tests.run_test(\n",
    "    \"validmind.data_validation.ClassImbalance\",\n",
    "    inputs={\n",
    "        \"dataset\": vm_raw_dataset,\n",
    "    },\n",
    "    params={\n",
    "        \"min_percent_threshold\": 10,\n",
    "    },\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc6_3_3_'></a>\n",
    "\n",
    "#### High Cardinality\n",
    "\n",
    "In this below case, the context specifies a risk-based criteria for the number of distinct values in categorical features.\n",
    "\n",
    "This helps the LLM to generate more nuanced and actionable insights, ensuring the descriptions are more relevant to your organization's policies."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_context = \"\"\"\n",
    "\n",
    "Acceptance Criteria:\n",
    "\n",
    "• Risk Levels for Distinct Values in Categorical Features:\n",
    "  - Low Risk: Each categorical column has fewer than 50 distinct values or less than 5% unique values relative to the total dataset size\n",
    "  - Medium Risk: Each categorical column has between 50 and 100 distinct values or between 5% and 10% unique values\n",
    "  - High Risk: Any categorical column has more than 100 distinct values or more than 10% unique values\n",
    "\n",
    "• Overall Requirement:\n",
    "  - All categorical columns must achieve at least Medium Risk status to pass\n",
    "\"\"\".strip()\n",
    "\n",
    "context = f\"\"\"\n",
    "{use_case_context}\n",
    "\n",
    "{test_context}\n",
    "\"\"\".strip()\n",
    "\n",
    "os.environ[\"VALIDMIND_LLM_DESCRIPTIONS_CONTEXT\"] = context"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the test-specific context set, generate a test description for the `HighCardinality` test for review:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vm.tests.run_test(\n",
    "    \"validmind.data_validation.HighCardinality\",\n",
    "    inputs={\n",
    "        \"dataset\": vm_raw_dataset,\n",
    "    },\n",
    "    params= {\n",
    "        \"num_threshold\": 100,\n",
    "        \"percent_threshold\": 0.1,\n",
    "        \"threshold_type\": \"percent\"\n",
    "        }\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc6_3_4_'></a>\n",
    "\n",
    "#### Missing Values\n",
    "\n",
    "Here, we use the test-specific context to establish differentiated risk thresholds across features.\n",
    "\n",
    "Rather than applying uniform criteria, the context allows for specific requirements for critical financial features (`CreditScore`, `Balance`, `Age`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_context = \"\"\"\n",
    "Test-Specific Context for Missing Values Analysis:\n",
    "\n",
    "Acceptance Criteria:\n",
    "\n",
    "• Risk Levels for Missing Values:\n",
    "  - Low Risk: Less than 1% missing values in any column\n",
    "  - Medium Risk: Between 1% and 5% missing values\n",
    "  - High Risk: More than 5% missing values\n",
    "\n",
    "• Feature-Specific Requirements:\n",
    "  - Critical Features (CreditScore, Balance, Age):\n",
    "    * Must maintain Low Risk status\n",
    "    * No missing values allowed\n",
    "  \n",
    "  - Secondary Features (Tenure, NumOfProducts, EstimatedSalary):\n",
    "    * Must achieve at least Medium Risk status\n",
    "    * Up to 3% missing values acceptable\n",
    "\n",
    "  - Categorical Features (Geography, Gender):\n",
    "    * Must achieve at least Medium Risk status\n",
    "    * Up to 5% missing values acceptable\n",
    "\"\"\".strip()\n",
    "\n",
    "context = f\"\"\"\n",
    "{use_case_context}\n",
    "\n",
    "{test_context}\n",
    "\"\"\".strip()\n",
    "\n",
    "os.environ[\"VALIDMIND_LLM_DESCRIPTIONS_CONTEXT\"] = context"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the test-specific context set, generate a test description for the `MissingValues` test for review:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vm.tests.run_test(\n",
    "    \"validmind.data_validation.MissingValues\",\n",
    "    inputs={\n",
    "        \"dataset\": vm_raw_dataset,\n",
    "    },\n",
    "    params= {\n",
    "        \"min_threshold\": 1\n",
    "        }\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc6_3_5_'></a>\n",
    "\n",
    "#### Unique Rows\n",
    "\n",
    "This example context establishes variable-specific thresholds based on business expectations.\n",
    "\n",
    "Rather than applying uniform criteria, it recognizes that high variability is expected in features like `EstimatedSalary` (>90%) and `Balance` (>50%), while enforcing strict limits on categorical features like `Geography` (<5 values), ensuring meaningful validation aligned with banking data characteristics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_context = \"\"\"\n",
    "\n",
    "Acceptance Criteria:\n",
    "\n",
    "• High-Variability Expected Features:\n",
    "  - EstimatedSalary: Must have >90% unique values\n",
    "  - Balance: Must have >50% unique values\n",
    "  - CreditScore: Must have between 5-10% unique values\n",
    "\n",
    "• Medium-Variability Features:\n",
    "  - Age: Should have between 0.5-2% unique values\n",
    "  - Tenure: Should have between 0.1-0.5% unique values\n",
    "\n",
    "• Low-Variability Features:\n",
    "  - Binary Features (HasCrCard, IsActiveMember, Gender, Exited): Must have exactly 2 unique values\n",
    "  - Geography: Must have fewer than 5 unique values\n",
    "  - NumOfProducts: Must have fewer than 10 unique values\n",
    "\n",
    "• Overall Requirements:\n",
    "  - Features must fall within their specified ranges to pass\n",
    "\"\"\".strip()\n",
    "\n",
    "context = f\"\"\"\n",
    "{use_case_context}\n",
    "\n",
    "{test_context}\n",
    "\"\"\".strip()\n",
    "\n",
    "os.environ[\"VALIDMIND_LLM_DESCRIPTIONS_CONTEXT\"] = context"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the test-specific context set, generate a test description for the `UniqueRows` test for review:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vm.tests.run_test(\n",
    "    \"validmind.data_validation.UniqueRows\",\n",
    "    inputs={\n",
    "        \"dataset\": vm_raw_dataset,\n",
    "    },\n",
    "    params= {\n",
    "        \"min_percent_threshold\": 1\n",
    "        }\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc6_3_6_'></a>\n",
    "\n",
    "#### Too Many Zero Values\n",
    "\n",
    "Here, test-specific context is used to provide meaning and expectations for different variables:\n",
    "\n",
    "- For instance, zero values in `Balance` and `Tenure` indicate risk, whereas zeros in binary variables like `HasCrCard` or `IsActiveMember` are expected.\n",
    "- This tailored context ensures that the analysis accurately reflects the business significance of zero values across different features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_context = \"\"\"\n",
    "\n",
    "Acceptance Criteria:\n",
    "- Numerical Features Only: Test evaluates only continuous numeric columns (Balance, Tenure), \n",
    "  excluding binary columns (HasCrCard, IsActiveMember)\n",
    "\n",
    "- Risk Level Thresholds for Balance and Tenure:\n",
    "  - High Risk: More than 5% zero values\n",
    "  - Medium Risk: Between 3% and 5% zero values\n",
    "  - Low Risk: Less than 3% zero values\n",
    "\n",
    "- Individual Column Requirements:\n",
    "  - Balance: Must be Low Risk (banking context requires accurate balance tracking)\n",
    "  - Tenure: Must be Low or Medium Risk (some zero values acceptable for new customers)\n",
    "\n",
    "• Overall Test Result: Test must achieve \"Pass\" status (Low Risk) for Balance, and at least Medium Risk for Tenure\n",
    "\n",
    "\"\"\".strip()\n",
    "\n",
    "context = f\"\"\"\n",
    "{use_case_context}\n",
    "\n",
    "{test_context}\n",
    "\"\"\".strip()\n",
    "\n",
    "os.environ[\"VALIDMIND_LLM_DESCRIPTIONS_CONTEXT\"] = context"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the test-specific context set, generate a test description for the `TooManyZeroValues` test for review:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vm.tests.run_test(\n",
    "    \"validmind.data_validation.TooManyZeroValues\",\n",
    "    inputs={\n",
    "        \"dataset\": vm_raw_dataset,\n",
    "    },\n",
    "    params= {\n",
    "        \"max_percent_threshold\": 0.03\n",
    "        }\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc6_3_7_'></a>\n",
    "\n",
    "#### IQR Outliers Table\n",
    "\n",
    "In this case, we use test-specific context to incorporate risk levels tailored to key variables, like `CreditScore`, `Age`, and `NumOfProducts`, that otherwise would not be considered for outlier analysis if we ran the test without context where all variables would be evaluated without any business criteria."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_context = \"\"\"\n",
    "\n",
    "Acceptance Criteria:\n",
    "- Risk Levels for Outliers:\n",
    "    - Low Risk: 0-50 outliers\n",
    "    - Medium Risk: 51-300 outliers\n",
    "    - High Risk: More than 300 outliers\n",
    "- Feature-Specific Requirements:\n",
    "    - CreditScore, Age, NumOfProducts: Must maintain Low Risk status to ensure data quality and model reliability\n",
    "\n",
    "\"\"\".strip()\n",
    "\n",
    "context = f\"\"\"\n",
    "{use_case_context}\n",
    "\n",
    "{test_context}\n",
    "\"\"\".strip()\n",
    "\n",
    "os.environ[\"VALIDMIND_LLM_DESCRIPTIONS_CONTEXT\"] = context"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the test-specific context set, generate a test description for the `IQROutliersTable` test for review:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vm.tests.run_test(\n",
    "    \"validmind.data_validation.IQROutliersTable\",\n",
    "    inputs={\n",
    "        \"dataset\": vm_raw_dataset,\n",
    "    },\n",
    "    params= {\n",
    "        \"threshold\": 1.5\n",
    "        }\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc6_3_8_'></a>\n",
    "\n",
    "#### Descriptive Statistics\n",
    "\n",
    "Test-specific context is used in this case to provide risk-based thresholds aligned with the bank's policy.\n",
    "\n",
    "For instance, `CreditScore` ranges of 550-850 are considered low risk based on standard credit assessment practices, while `Balance` thresholds reflect typical retail banking ranges."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_context = \"\"\"\n",
    "\n",
    "Acceptance Criteria:\n",
    "\n",
    "• CreditScore:\n",
    "  - Low Risk: 550-850\n",
    "  - Medium Risk: 450-549\n",
    "  - High Risk: <450 or missing\n",
    "  - Justification: Banking standards require reliable credit assessment\n",
    "\n",
    "• Age:\n",
    "  - Low Risk: 18-75\n",
    "  - Medium Risk: 76-85\n",
    "  - High Risk: >85 or <18\n",
    "  - Justification: Core banking demographic with age-appropriate products\n",
    "\n",
    "• Balance:\n",
    "  - Low Risk: 0-200,000\n",
    "  - Medium Risk: 200,001-250,000\n",
    "  - High Risk: >250,000\n",
    "  - Justification: Typical retail banking balance ranges\n",
    "\n",
    "• Tenure:\n",
    "  - Low Risk: 1-10 years\n",
    "  - Medium Risk: <1 year\n",
    "  - High Risk: 0 or >10 years\n",
    "  - Justification: Expected customer relationship duration\n",
    "\n",
    "• EstimatedSalary:\n",
    "  - Low Risk: 25,000-150,000\n",
    "  - Medium Risk: 150,001-200,000\n",
    "  - High Risk: <25,000 or >200,000\n",
    "  - Justification: Typical income ranges for retail banking customers\n",
    "\n",
    "\"\"\".strip()\n",
    "\n",
    "context = f\"\"\"\n",
    "{use_case_context}\n",
    "\n",
    "{test_context}\n",
    "\"\"\".strip()\n",
    "\n",
    "os.environ[\"VALIDMIND_LLM_DESCRIPTIONS_CONTEXT\"] = context"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the test-specific context set, generate a test description for the `DescriptiveStatistics` test for review:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vm.tests.run_test(\n",
    "    \"validmind.data_validation.DescriptiveStatistics\",\n",
    "    inputs={\n",
    "        \"dataset\": vm_raw_dataset,\n",
    "    },\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc6_3_9_'></a>\n",
    "\n",
    "#### Pearson Correlation Matrix\n",
    "\n",
    "For this test, the context provides meaningful correlation ranges between specific variable pairs based on business criteria.\n",
    "\n",
    "For example, while a general correlation analysis might flag any correlation above 0.7 as concerning, the test-specific context specifies that `Balance` and `NumOfProducts` should maintain a negative correlation between -0.4 and 0, reflecting expected banking relationships."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_context = \"\"\"\n",
    "\n",
    "Acceptance Criteria:\n",
    "\n",
    "• Target Variable Correlations (Exited):\n",
    "  - Must show correlation coefficients between ±0.1 and ±0.3 with Age, CreditScore, and Balance\n",
    "  - Should not exceed ±0.2 correlation with other features\n",
    "  - Justification: Ensures predictive power while avoiding target leakage\n",
    "\n",
    "• Feature Correlations:\n",
    "  - Balance & NumOfProducts: Must maintain correlation between -0.4 and 0\n",
    "  - Age & Tenure: Should show positive correlation between 0.1 and 0.3\n",
    "  - CreditScore & Balance: Should maintain correlation between 0.1 and 0.3\n",
    "\n",
    "• Binary Feature Correlations:\n",
    "  - HasCreditCard & IsActiveMember: Must not exceed ±0.15 correlation\n",
    "  - Binary features should not show strong correlations (>±0.2) with continuous features\n",
    "\n",
    "• Overall Requirement:\n",
    "  - No feature pair should exceed ±0.7 correlation to avoid multicollinearity\n",
    "\n",
    "\"\"\".strip()\n",
    "\n",
    "context = f\"\"\"\n",
    "{use_case_context}\n",
    "\n",
    "{test_context}\n",
    "\"\"\".strip()\n",
    "\n",
    "os.environ[\"VALIDMIND_LLM_DESCRIPTIONS_CONTEXT\"] = context"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the test-specific context set, generate a test description for the `PearsonCorrelationMatrix` test for review:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vm.tests.run_test(\n",
    "    \"validmind.data_validation.PearsonCorrelationMatrix\",\n",
    "    inputs={\n",
    "        \"dataset\": vm_raw_dataset,\n",
    "    },\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc6_4_'></a>\n",
    "\n",
    "### Add test-specific context using the docstring\n",
    "\n",
    "Another way to customize test result descriptions is to include explicit instructions in the test docstring:\n",
    "\n",
    "- Unlike the environment variable methods above which require runtime configuration and is best for dynamic customization across multiple levels (global, test suite, test-specific), modifying the docstring permanently embeds instructions within the test definition itself.\n",
    "- This docstring approach is ideal for tests with consistent reporting requirements that should persist across environments, ensuring standardized outputs regardless of external configuration settings.\n",
    "\n",
    "In the following example, we demonstrate using post-processing functions to dynamically modify the docstring at runtime, which is useful for experimentation in notebooks. However, users can alternatively hardcode these same instructions directly in the test's docstring if they want these customizations to be permanently part of the test definition without requiring additional runtime code. **Use this method when you want instructions to remain an intrinsic part of the test's definition,** eliminating the need to repeatedly set environment variables in different execution contexts.\n",
    "\n",
    "We'll implement a custom test with a default docstring that follows the ValidMind docstring structure. First, we will run this custom test with the default description:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "os.environ[\"VALIDMIND_LLM_DESCRIPTIONS_CONTEXT_ENABLED\"] = \"0\"\n",
    "\n",
    "@vm.test(\"my_custom_tests.MissingValues\")\n",
    "def MissingValues(dataset, min_threshold = 1):\n",
    "    \"\"\"\n",
    "    Evaluates dataset quality by ensuring missing value ratio across all features does not exceed a set threshold.\n",
    "\n",
    "    ### Purpose\n",
    "\n",
    "    The Missing Values test is designed to evaluate the quality of a dataset by measuring the number of missing values\n",
    "    across all features. The objective is to ensure that the ratio of missing data to total data is less than a\n",
    "    predefined threshold, defaulting to 1, in order to maintain the data quality necessary for reliable predictive\n",
    "    strength in a machine learning model.\n",
    "\n",
    "    ### Test Mechanism\n",
    "\n",
    "    The mechanism for this test involves iterating through each column of the dataset, counting missing values\n",
    "    (represented as NaNs), and calculating the percentage they represent against the total number of rows. The test\n",
    "    then checks if these missing value counts are less than the predefined `min_threshold`. The results are shown in a\n",
    "    table summarizing each column, the number of missing values, the percentage of missing values in each column, and a\n",
    "    Pass/Fail status based on the threshold comparison.\n",
    "\n",
    "    ### Signs of High Risk\n",
    "\n",
    "    - When the number of missing values in any column exceeds the `min_threshold` value.\n",
    "    - Presence of missing values across many columns, leading to multiple instances of failing the threshold.\n",
    "\n",
    "    ### Strengths\n",
    "\n",
    "    - Quick and granular identification of missing data across each feature in the dataset.\n",
    "    - Provides an effective and straightforward means of maintaining data quality, essential for constructing efficient\n",
    "    machine learning models.\n",
    "\n",
    "    ### Limitations\n",
    "\n",
    "    - Does not suggest the root causes of the missing values or recommend ways to impute or handle them.\n",
    "    - May overlook features with significant missing data but still less than the `min_threshold`, potentially\n",
    "    impacting the model.\n",
    "    - Does not account for data encoded as values like \"-999\" or \"None,\" which might not technically classify as\n",
    "    missing but could bear similar implications.\n",
    "    \"\"\"\n",
    "    df = dataset.df\n",
    "    missing = df.isna().sum()\n",
    "\n",
    "    return (\n",
    "        [\n",
    "            {\n",
    "                \"Column\": col,\n",
    "                \"Number of Missing Values\": missing[col],\n",
    "                \"Percentage of Missing Values (%)\": missing[col] / df.shape[0] * 100,\n",
    "                \"Pass/Fail\": \"Pass\" if missing[col] < min_threshold else \"Fail\",\n",
    "            }\n",
    "            for col in missing.index\n",
    "        ],\n",
    "        all(missing[col] < min_threshold for col in missing.index),\n",
    "    )\n",
    "\n",
    "vm.tests.run_test(\n",
    "    \"my_custom_tests.MissingValues\",\n",
    "    inputs={\"dataset\": vm_raw_dataset},\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's append custom instructions to the test docstring by using a post-processing function that modifies the default docstring before rerunning the test:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "from validmind.vm_models.result import TestResult\n",
    "\n",
    "# This function will append the instructions to the end of the docstring\n",
    "def add_instructions(result: TestResult): \n",
    "    result.doc += \"\"\"\\n\\nINSTRUCTIONS: \n",
    "    - Generate 5 Key insights.\n",
    "    - Add the following note at the end of the generated output: '*NOTE: This is a sample of the data, for the full data results please look in the appendix*'\n",
    "    \"\"\"\n",
    "    return result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You’ll notice that the description generated by the LLM is now updated to reflect the appended instructions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vm.tests.run_test(\n",
    "    \"my_custom_tests.MissingValues\",\n",
    "    inputs={\"dataset\": vm_raw_dataset},\n",
    "    post_process_fn=add_instructions,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we are happy with the customization, we can now hardcode the instructions into our docstring, so they become a permanent part of the test definition and persist across any environment where the test is executed:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "os.environ[\"VALIDMIND_LLM_DESCRIPTIONS_CONTEXT_ENABLED\"] = \"0\"\n",
    "\n",
    "@vm.test(\"my_custom_tests.MissingValues\")\n",
    "def MissingValues(dataset, min_threshold = 1):\n",
    "    \"\"\"\n",
    "    Evaluates dataset quality by ensuring missing value ratio across all features does not exceed a set threshold.\n",
    "\n",
    "    ### Purpose\n",
    "\n",
    "    The Missing Values test is designed to evaluate the quality of a dataset by measuring the number of missing values\n",
    "    across all features. The objective is to ensure that the ratio of missing data to total data is less than a\n",
    "    predefined threshold, defaulting to 1, in order to maintain the data quality necessary for reliable predictive\n",
    "    strength in a machine learning model.\n",
    "\n",
    "    ### Test Mechanism\n",
    "\n",
    "    The mechanism for this test involves iterating through each column of the dataset, counting missing values\n",
    "    (represented as NaNs), and calculating the percentage they represent against the total number of rows. The test\n",
    "    then checks if these missing value counts are less than the predefined `min_threshold`. The results are shown in a\n",
    "    table summarizing each column, the number of missing values, the percentage of missing values in each column, and a\n",
    "    Pass/Fail status based on the threshold comparison.\n",
    "\n",
    "    ### Signs of High Risk\n",
    "\n",
    "    - When the number of missing values in any column exceeds the `min_threshold` value.\n",
    "    - Presence of missing values across many columns, leading to multiple instances of failing the threshold.\n",
    "\n",
    "    ### Strengths\n",
    "\n",
    "    - Quick and granular identification of missing data across each feature in the dataset.\n",
    "    - Provides an effective and straightforward means of maintaining data quality, essential for constructing efficient\n",
    "    machine learning models.\n",
    "\n",
    "    ### Limitations\n",
    "\n",
    "    - Does not suggest the root causes of the missing values or recommend ways to impute or handle them.\n",
    "    - May overlook features with significant missing data but still less than the `min_threshold`, potentially\n",
    "    impacting the model.\n",
    "    - Does not account for data encoded as values like \"-999\" or \"None,\" which might not technically classify as\n",
    "    missing but could bear similar implications.\n",
    "\n",
    "    INSTRUCTIONS: \n",
    "    - Generate 5 Key insights.\n",
    "    - Add the following note at the end of the generated output: '*NOTE: This is a sample of the data, for the full data results please look in the appendix*'\n",
    "    \"\"\"\n",
    "    df = dataset.df\n",
    "    missing = df.isna().sum()\n",
    "\n",
    "    return (\n",
    "        [\n",
    "            {\n",
    "                \"Column\": col,\n",
    "                \"Number of Missing Values\": missing[col],\n",
    "                \"Percentage of Missing Values (%)\": missing[col] / df.shape[0] * 100,\n",
    "                \"Pass/Fail\": \"Pass\" if missing[col] < min_threshold else \"Fail\",\n",
    "            }\n",
    "            for col in missing.index\n",
    "        ],\n",
    "        all(missing[col] < min_threshold for col in missing.index),\n",
    "    )\n",
    "\n",
    "vm.tests.run_test(\n",
    "    \"my_custom_tests.MissingValues\",\n",
    "    inputs={\"dataset\": vm_raw_dataset},\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc7_'></a>\n",
    "\n",
    "## Best practices for adding custom context\n",
    "\n",
    "When working with test result descriptions, it's often useful to provide custom instructions that guide how a test should be interpreted. There are two main ways to add this context: through environment variables or by modifying the test's docstring. Each method has different use cases, levels of persistence, and scopes of influence. Understanding when to use each can help ensure clarity, maintainability, and consistency in your testing workflows. \n",
    "\n",
    "The **test’s docstring** is a natural place to embed instructions that are closely tied to the purpose and interpretation of that specific test. There are two ways to leverage docstrings:\n",
    "\n",
    "- Hardcoded instructions can be written directly into the test function’s docstring in the source code. These instructions become a permanent part of the test definition and will persist across all environments where the test is run. This approach is ideal when you want clear, consistent guidance that always travels with the test.\n",
    "- Dynamic modification of the docstring at runtime allows you to append instructions using a post-processing function. This is useful in interactive or experimental settings—such as notebooks—where you may want to fine-tune the test’s description temporarily. These instructions are local to the current test and do not affect others\n",
    "\n",
    "While docstrings are localized to individual tests, **environment variables** offer a broader mechanism for injecting context at runtime. This method is best used when you want to apply the same guidance across multiple tests in a session. For example, you might define a high-level context once and have it apply globally throughout a suite of tests. However, because environment variables persist beyond a single test, they can unintentionally influence the behavior of subsequent tests unless explicitly overridden or cleared. This global scope is powerful, but it requires careful handling to avoid unexpected side effects.\n",
    "\n",
    "**Choosing the right approach**\n",
    "- Use hardcoded docstring instructions when you want test guidance to be a permanent part of the test definition, ensuring consistency across environments.\n",
    "- Use docstring post-processing when you need flexibility for local or temporary customization, particularly in experimental or development settings.\n",
    "- Use environment variables to apply a shared context across multiple tests, but be mindful that the configuration will persist unless you reset it.\n",
    "\n",
    "This tiered approach provides both precision and flexibility—letting you decide whether context should live inside the test, be generated dynamically at runtime, or apply more broadly across test runs."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "ValidMind Library",
   "language": "python",
   "name": "validmind"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.15"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
