{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Automate news summarization using LLMs\n",
    "\n",
    "Document a LLM-based text summarization model of news using the [CNN DailyMail](https://huggingface.co/datasets/cnn_dailymail) sample dataset from HuggingFace with the ValidMind Library.\n",
    "\n",
    "As part of the notebook, you will learn how to develop an text summarization model while exploring how the documentation process works:\n",
    "\n",
    "- Initializing the ValidMind Library\n",
    "- Loading a sample dataset provided by the library to develop a text summarization model using LLMs\n",
    "- Running a ValidMind test suite to quickly generate documention about the data and model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Use case\n",
    "\n",
    "The purpose of this notebook is to showcase how to document an automated news summarization system using a Large Language Model (LLM). This AI system leverages a large language model (LLM) to process and condense web-based news articles into concise summaries. \n",
    "\n",
    "**Data Sources**\n",
    "\n",
    "The CNN/DailyMail Dataset is a collection tailored for text summarization, containing over 300,000 news articles from two significant English-speaking regions, the US and the UK. Each row comprises an article, a highlight section, and a unique ID. The highlights in the dataset are summaries that have been written by the original journalists. The CNN articles were written between April 2007 and April 2015. The Daily Mail articles were written between June 2010 and April 2015.\n",
    "\n",
    "The original dataset includes pre-divided splits: train, validation, and test. In this demo, as we are not training a LLM in the traditional machine learning sense but rather using prompt engineering to guide the LLM to function as a text summarizer, we do not adhere to the conventional distinction between training and test datasets. Therefore, we exclusively utilize the test dataset, applying it as a validation or \"gold\" standard to evaluate the effectiveness of our summarization through prompt engineering.\n",
    "\n",
    "**Workflow**\n",
    "\n",
    "The workflow comprises four primary stages, starting with article selection, where articles from the test dataset are chosen. This is followed by prompt engineering, where a prompt is crafted to communicate the summarization task to the LLM. In the summarization stage, the prompt is input into the LLM, which then produces summaries based on the article content. The final stage involves LLM response evaluation, where the summaries generated by the LLM are measured against the original journalist-authored highlights to evaluate the summarization quality."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## About ValidMind\n",
    "\n",
    "ValidMind's suite of tools enables organizations to identify, document, and manage model risks for all types of models, including AI/ML models, LLMs, and statistical models. As a model developer, you use the ValidMind Library to automate documentation and validation tests, and then use the ValidMind Platform to collaborate on model documentation. Together, these products simplify model risk management, facilitate compliance with regulations and institutional standards, and enhance collaboration between yourself and model validators.\n",
    "\n",
    "If this is your first time trying out ValidMind, you can make use of the following resources alongside this notebook:\n",
    "\n",
    "- [Get started](https://docs.validmind.ai/get-started/get-started.html) — The basics, including key concepts, and how our products work\n",
    "- [ValidMind Library](https://docs.validmind.ai/developer/validmind-library.html) — The path for developers, more code samples, and our developer reference\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Before you begin\n",
    "\n",
    "<div class=\"alert alert-block alert-info\" style=\"background-color: #B5B5B510; color: black; border: 1px solid #083E44; border-left-width: 5px; box-shadow: 2px 2px 4px rgba(0, 0, 0, 0.2);border-radius: 5px;\"><span style=\"color: #083E44;\"><b>For access to all features available in this notebook, create a free ValidMind account.</b></span>\n",
    "<br></br>\n",
    "Signing up is FREE — <a href=\"https://docs.validmind.ai/guide/configuration/register-with-validmind.html\" style=\"color: #DE257E;\"><b>Register with ValidMind</b></a></div>\n",
    "\n",
    "If you encounter errors due to missing modules in your Python environment, install the modules with `pip install`, and then re-run the notebook. For more help, refer to [Installing Python Modules](https://docs.python.org/3/installing/index.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Install the ValidMind Library\n",
    "\n",
    "To install the library:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install -q validmind"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Initialize the ValidMind Library\n",
    "\n",
    "ValidMind generates a unique _code snippet_ for each registered model to connect with your developer environment. You initialize the ValidMind Library with this code snippet, which ensures that your documentation and tests are uploaded to the correct model when you run the notebook.\n",
    "\n",
    "### Get your code snippet\n",
    "\n",
    "1. In a browser, [log in to ValidMind](https://docs.validmind.ai/guide/configuration/log-in-to-validmind.html).\n",
    "\n",
    "2. In the left sidebar, navigate to **Model Inventory** and click **+ Register Model**.\n",
    "\n",
    "3. Enter the model details and click **Continue**. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/register-models-in-inventory.html))\n",
    "\n",
    "   For example, to register a model for use with this notebook, select:\n",
    "\n",
    "   - Documentation template: `LLM-based Text Summarization`\n",
    "   - Use case: `Marketing/Sales - Sales/Prospecting`\n",
    "\n",
    "   You can fill in other options according to your preference.\n",
    "\n",
    "4. Go to **Getting Started** and click **Copy snippet to clipboard**.\n",
    "\n",
    "Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load your model identifier credentials from an `.env` file\n",
    "\n",
    "%load_ext dotenv\n",
    "%dotenv .env\n",
    "\n",
    "# Or replace with your code snippet\n",
    "\n",
    "import validmind as vm\n",
    "\n",
    "vm.init(\n",
    "    # api_host=\"...\",\n",
    "    # api_key=\"...\",\n",
    "    # api_secret=\"...\",\n",
    "    # model=\"...\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Initialize the Python environment\n",
    "\n",
    "Next, let's import the necessary libraries and set up your Python environment for data analysis:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Install the `datasets` library from huggingface\n",
    "%pip install -q datasets\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preview the documentation template\n",
    "\n",
    "A template predefines sections for your model documentation and provides a general outline to follow, making the documentation process much easier.\n",
    "\n",
    "You will upload documentation and test results into this template later on. For now, take a look at the structure that the template provides with the `vm.preview_template()` function from the ValidMind library and note the empty sections:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vm.preview_template()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load the sample dataset\n",
    "\n",
    "The sample dataset used here is provided by the ValidMind library. To be able to use it, you need to import the dataset and load it into a pandas [DataFrame](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.html), a two-dimensional tabular data structure that makes use of rows and columns:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import the sample dataset from the library\n",
    "from validmind.datasets.nlp import cnn_dailymail\n",
    "\n",
    "print(\n",
    "    f\"Loaded demo dataset with: \\n\\n\\t• Target column: '{cnn_dailymail.target_column}' \"\n",
    "    f\"\\n\\t• Input text column: {cnn_dailymail.text_column} \"\n",
    "    f\"\\n\\t• Prediction columns: '{cnn_dailymail.t5_prediction}', '{cnn_dailymail.gpt_35_prediction_column}'\"\n",
    ")\n",
    "\n",
    "\n",
    "train_df, test_df = cnn_dailymail.load_data(source=\"offline\", dataset_size=\"100\")\n",
    "\n",
    "# Display the first few rows of the dataframe to check the loaded data. Ignore the \"bert_embedding_model_prediction\" column\n",
    "cnn_dailymail.display_nice(train_df.drop(\"bert_embedding_model_prediction\", axis=1).head())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Document the model\n",
    "\n",
    "As part of documenting the model with the ValidMind Library, you need to preprocess the raw dataset, initialize some training and test datasets, initialize a model object you can use for testing, and then run the full suite of tests."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Setup the Large Language Model (LLM)\n",
    "\n",
    "This section prepares our environment to use OpenAI's Large Language Model by setting up the API key and defining a function to call the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "import dotenv\n",
    "import nltk\n",
    "\n",
    "dotenv.load_dotenv()\n",
    "nltk.download('stopwords')\n",
    "\n",
    "# os.environ[\"OPENAI_API_KEY\"] = \"sk-...\"\n",
    "\n",
    "if os.getenv(\"OPENAI_API_KEY\") is None:\n",
    "    raise Exception(\"OPENAI_API_KEY not found\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from openai import OpenAI\n",
    "\n",
    "model = OpenAI()\n",
    "\n",
    "\n",
    "def call_model(prompt):\n",
    "    return (\n",
    "        model.chat.completions.create(\n",
    "            model=\"gpt-3.5-turbo\",\n",
    "            messages=[\n",
    "                {\"role\": \"user\", \"content\": prompt},\n",
    "            ],\n",
    "        )\n",
    "        .choices[0]\n",
    "        .message.content\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Setup up the Prompt\n",
    "\n",
    "In this section, we construct a structured prompt template designed to guide the AI in summarizing the CNN Daily news. The template emphasizes the AI's role as an expert in parsing and condensing news information. It instructs the AI to focus on the article's core content, avoiding assumptions or external data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "prompt_template = \"\"\"\n",
    "You are an AI with expertise in summarizing financial news.\n",
    "Your task is to provide a concise summary of the specific news article provided below.\n",
    "Before proceeding, take a moment to understand the context and nuances of the financial terminology used in the article.\n",
    "\n",
    "Article to Summarize:\n",
    "\n",
    "```\n",
    "{article}\n",
    "```\n",
    "\n",
    "Please respond with a concise summary of the article's main points.\n",
    "Ensure that your summary is based on the content of the article and not on external information or assumptions.\n",
    "\"\"\".strip()\n",
    "\n",
    "prompt_variables = [\"article\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Initialize the ValidMind datasets\n",
    "\n",
    "Before you can run tests, you must first initialize a ValidMind dataset object using the [`init_dataset`](https://docs.validmind.ai/validmind/validmind.html#init_dataset) function from the ValidMind (`vm`) module.\n",
    "\n",
    "This function takes a number of arguments:\n",
    "\n",
    "- `dataset` — the raw dataset that you want to provide as input to tests\n",
    "- `input_id` - a unique identifier that allows tracking what inputs are used when running each individual test\n",
    "- `target_column` — a required argument if tests require access to true values. This is the name of the target column in the dataset\n",
    "\n",
    "With all datasets ready, you can now initialize training and test datasets (`train_df` and `test_df`) created earlier into their own dataset objects using [`vm.init_dataset()`](https://docs.validmind.ai/validmind/validmind.html#init_dataset):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from validmind.models import FoundationModel, Prompt\n",
    "\n",
    "vm_test_ds = vm.init_dataset(\n",
    "    dataset=test_df,\n",
    "    input_id=\"test_dataset\",\n",
    "    text_column=\"article\",\n",
    "    target_column=\"highlights\",\n",
    ")\n",
    "\n",
    "vm_model = vm.init_model(\n",
    "    model=FoundationModel(\n",
    "        predict_fn=call_model,\n",
    "        prompt=Prompt(\n",
    "            template=prompt_template,\n",
    "            variables=prompt_variables,\n",
    "        ),\n",
    "    ),\n",
    "    input_id=\"gpt_35\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Assign predictions to the datasets\n",
    "\n",
    "We can now use the assign_predictions() method from the Dataset object to link existing predictions to any model. If no prediction values are passed, the method will compute predictions automatically:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Assign pre-computed model predictions to the test dataset\n",
    "vm_test_ds.assign_predictions(vm_model, prediction_column=\"gpt_35_prediction\")\n",
    "\n",
    "print(vm_test_ds)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data validation\n",
    "\n",
    "This section focuses on performing a series of data description tests to gain insights into the basic characteristics of our text data. The goal of data description in this use case is verifying that the data meets certain standards and criteria before it is used for text summarization tasks. We conduct the follwoing NLP data quality tests:\n",
    "\n",
    "- *Duplicates*: Check for duplicate articles in the dataset.\n",
    "- *Text Description*: Assess the general context and provide a summary of the dataset.\n",
    "- *Common Words*: Determine the most frequently occurring words that could indicate key themes.\n",
    "- *Punctuations*: Analyze punctuation patterns to understand sentence structures and emphases.\n",
    "- *Stop Words*: Identify and remove common stopwords to clarify the significant textual elements.\n",
    "- *Language Detection*: Verify the language of the dataset to ensure it is consistent.\n",
    "- *Toxicity*: Evaluate the presence of toxic language in the dataset.\n",
    "- *Polarity and Subjectivity*: Measure the sentiment of the dataset to understand the overall tone.\n",
    "- *Sentiment*: Analyze the sentiment of the dataset to determine the overall mood.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test = vm.tests.run_test(\n",
    "    \"validmind.data_validation.Duplicates\",\n",
    "    inputs={\n",
    "        \"dataset\": vm_test_ds,\n",
    "    },\n",
    ")\n",
    "test.log()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test = vm.tests.run_test(\n",
    "    \"validmind.data_validation.nlp.TextDescription\",\n",
    "    inputs={\n",
    "        \"dataset\": vm_test_ds,\n",
    "    },\n",
    ")\n",
    "test.log()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test = vm.tests.run_test(\n",
    "    \"validmind.data_validation.nlp.CommonWords\",\n",
    "    inputs={\n",
    "        \"dataset\": vm_test_ds,\n",
    "    },\n",
    ")\n",
    "test.log()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test = vm.tests.run_test(\n",
    "    \"validmind.data_validation.nlp.Punctuations\",\n",
    "    inputs={\n",
    "        \"dataset\": vm_test_ds,\n",
    "    },\n",
    "    params = {\n",
    "        \"count_mode\": \"word\"\n",
    "    }\n",
    ")\n",
    "test.log()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test = vm.tests.run_test(\n",
    "    \"validmind.data_validation.nlp.StopWords\",\n",
    "    inputs={\n",
    "        \"dataset\": vm_test_ds,\n",
    "    },\n",
    ")\n",
    "test.log()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test = vm.tests.run_test(\n",
    "    \"validmind.data_validation.nlp.LanguageDetection\",\n",
    "    inputs={\n",
    "        \"dataset\": vm_test_ds,\n",
    "    },\n",
    ")\n",
    "test.log()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test = vm.tests.run_test(\n",
    "    \"validmind.data_validation.nlp.Toxicity\",\n",
    "    inputs={\n",
    "        \"dataset\": vm_test_ds,\n",
    "    },\n",
    ")\n",
    "test.log()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test = vm.tests.run_test(\n",
    "    \"validmind.data_validation.nlp.PolarityAndSubjectivity\",\n",
    "    inputs={\n",
    "        \"dataset\": vm_test_ds,\n",
    "    },\n",
    ")\n",
    "test.log()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test = vm.tests.run_test(\n",
    "    \"validmind.data_validation.nlp.Sentiment\",\n",
    "    inputs={\n",
    "        \"dataset\": vm_test_ds,\n",
    "    },\n",
    ")\n",
    "test.log()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Prompt Validation\n",
    "\n",
    "This section conducts a critical analysis of prompts to ensure their effectiveness when interacting with AI models. It involves systematic checks across several dimensions to enhance the quality of the interaction between the user and the AI:\n",
    "\n",
    "- *Bias*: Evaluate prompts for impartiality.\n",
    "- *Clarity*: Confirm the prompts are clearly understood.\n",
    "- *Conciseness*: Verify that the prompts are brief and concise.\n",
    "- *Delimitation*: Check the boundaries and extent of prompts.\n",
    "- *Negative Instruction*: Review prompts for any negative phrasing that could be misconstrued.\n",
    "- *Specificity*: Assess prompts for detailed and precise instructions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test = vm.tests.run_test(\n",
    "    \"validmind.prompt_validation.Bias\",\n",
    "    inputs={\n",
    "        \"dataset\": vm_test_ds,\n",
    "        \"model\": vm_model,\n",
    "    },\n",
    ")\n",
    "test.log()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test = vm.tests.run_test(\n",
    "    \"validmind.prompt_validation.Clarity\",\n",
    "    inputs={\n",
    "        \"dataset\": vm_test_ds,\n",
    "        \"model\": vm_model,\n",
    "    },\n",
    ")\n",
    "test.log()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test = vm.tests.run_test(\n",
    "    \"validmind.prompt_validation.Conciseness\",\n",
    "    inputs={\n",
    "        \"dataset\": vm_test_ds,\n",
    "        \"model\": vm_model,\n",
    "    },\n",
    ")\n",
    "test.log()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test = vm.tests.run_test(\n",
    "    \"validmind.prompt_validation.Delimitation\",\n",
    "    inputs={\n",
    "        \"dataset\": vm_test_ds,\n",
    "        \"model\": vm_model,\n",
    "    },\n",
    ")\n",
    "test.log()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test = vm.tests.run_test(\n",
    "    \"validmind.prompt_validation.NegativeInstruction\",\n",
    "    inputs={\n",
    "        \"dataset\": vm_test_ds,\n",
    "        \"model\": vm_model,\n",
    "    },\n",
    ")\n",
    "test.log()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test = vm.tests.run_test(\n",
    "    \"validmind.prompt_validation.Specificity\",\n",
    "    inputs={\n",
    "        \"dataset\": vm_test_ds,\n",
    "        \"model\": vm_model,\n",
    "    },\n",
    ")\n",
    "test.log()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model Validation\n",
    "\n",
    "This section is dedicated to the assessment of the AI model's understanding and processing of language data. It involves running various model performance tests, ensuring the model's output is as expected and reliable."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Run model performance tests\n",
    "\n",
    "Here we measure the model's linguistic performance across various metrics, including:\n",
    "\n",
    "- *Token Disparity*: Examine the distribution of token usage.\n",
    "- *Rouge Metrics*: Use Recall-Oriented Understudy for Gisting Evaluation to assess the summary quality.\n",
    "- *Bert Score*: Implement BERT-based evaluations of token similarity.\n",
    "- *Contextual Recall*: Test the model's ability to recall contextual information.\n",
    "- *Bleu Score*: Evaluate the quality of machine translation.\n",
    "- *Meteor Score*: Measure translation hypothesis against reference translations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test = vm.tests.run_test(\n",
    "    \"validmind.model_validation.TokenDisparity\",\n",
    "    inputs={\n",
    "        \"dataset\": vm_test_ds,\n",
    "        \"model\": \"gpt_35\",\n",
    "    },\n",
    ")\n",
    "test.log()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test = vm.tests.run_test(\n",
    "    \"validmind.model_validation.RougeScore\",\n",
    "    inputs={\n",
    "        \"dataset\": vm_test_ds,\n",
    "        \"model\": vm_model,\n",
    "    },\n",
    "    params={\n",
    "        \"metric\": \"rouge-1\",\n",
    "    },\n",
    ")\n",
    "test.log()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test = vm.tests.run_test(\n",
    "    \"validmind.model_validation.BertScore\",\n",
    "    inputs={\n",
    "        \"dataset\": vm_test_ds,\n",
    "        \"model\": vm_model,\n",
    "    },\n",
    ")\n",
    "test.log()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test = vm.tests.run_test(\n",
    "    \"validmind.model_validation.ContextualRecall\",\n",
    "    inputs={\n",
    "        \"dataset\": vm_test_ds,\n",
    "        \"model\": vm_model,\n",
    "    },\n",
    ")\n",
    "test.log()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test = vm.tests.run_test(\n",
    "    \"validmind.model_validation.BleuScore\",\n",
    "    inputs={\n",
    "        \"dataset\": vm_test_ds,\n",
    "        \"model\": vm_model,\n",
    "    },\n",
    ")\n",
    "test.log()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test = vm.tests.run_test(\n",
    "    \"validmind.model_validation.MeteorScore\",\n",
    "    inputs={\n",
    "        \"dataset\": vm_test_ds,\n",
    "        \"model\": vm_model,\n",
    "    },\n",
    ")\n",
    "test.log()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Run bias and toxicity tests\n",
    "\n",
    "The focus of this subsection is on identifying any potential bias or toxicity in the model's language processing. We conduct:\n",
    "\n",
    "- *Toxicity Score*: Quantify the degree of toxicity in content generated by the model.\n",
    "- *Toxicity Histogram*: Visualize the distribution of toxicity scores.\n",
    "- *Regard Score*: Assess the model's language for indications of respect or disrespect.\n",
    "- *Regard Histogram*: Plot the frequencies of different levels of regard to identify patterns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test = vm.tests.run_test(\n",
    "    \"validmind.model_validation.ToxicityScore\",\n",
    "    inputs={\n",
    "        \"dataset\": vm_test_ds,\n",
    "        \"model\": vm_model,\n",
    "    },\n",
    ")\n",
    "test.log()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test = vm.tests.run_test(\n",
    "    \"validmind.model_validation.RegardScore\",\n",
    "    inputs={\n",
    "        \"dataset\": vm_test_ds,\n",
    "        \"model\": vm_model,\n",
    "    },\n",
    ")\n",
    "test.log()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Next steps\n",
    "\n",
    "You can look at the results of this test suite right in the notebook where you ran the code, as you would expect. But there is a better way: view the prompt validation test results as part of your model documentation right in the ValidMind Platform:\n",
    "\n",
    "1. In the ValidMind Platform, go to the **Documentation** page for the model you registered earlier. ([Need more help?](https://docs.validmind.ai/guide/model-documentation/working-with-model-documentation.html)\n",
    "\n",
    "2. Expand **2. Data Preparation** or **3. Model Development** to review all test results.\n",
    "\n",
    "What you can see now is a more easily consumable version of the prompt validation testing you just performed, along with other parts of your model documentation that still need to be completed.\n",
    "\n",
    "If you want to learn more about where you are in the model documentation process, take a look at our documentation on the [ValidMind Library](https://docs.validmind.ai/developer/validmind-library.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Upgrade ValidMind\n",
    "\n",
    "<div class=\"alert alert-block alert-info\" style=\"background-color: #B5B5B510; color: black; border: 1px solid #083E44; border-left-width: 5px; box-shadow: 2px 2px 4px rgba(0, 0, 0, 0.2);border-radius: 5px;\">After installing ValidMind, you’ll want to periodically make sure you are on the latest version to access any new features and other enhancements.</div>\n",
    "\n",
    "Retrieve the information for the currently installed version of ValidMind:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip show validmind"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If the version returned is lower than the version indicated in our [production open-source code](https://github.com/validmind/validmind-library/blob/prod/validmind/__version__.py), restart your notebook and run:\n",
    "\n",
    "```bash\n",
    "%pip install --upgrade validmind\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You may need to restart your kernel after running the upgrade package for changes to be applied."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
