{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# RAG Model Documentation Demo\n",
    "\n",
    "In this notebook, we are going to implement a simple RAG Model for automating the process of answering RFP questions using GenAI. We will see how we can initialize an embedding model, a retrieval model and a generator model with LangChain components and use them within the ValidMind Library to run tests against them. Finally, we will see how we can put them together in a Pipeline and run that to get e2e results and run tests against that."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc2_'></a>\n",
    "\n",
    "## About ValidMind\n",
    "\n",
    "ValidMind is a suite of tools for managing model risk, including risk associated with AI and statistical models.\n",
    "\n",
    "You use the ValidMind Library to automate documentation and validation tests, and then use the ValidMind Platform to collaborate on model documentation. Together, these products simplify model risk management, facilitate compliance with regulations and institutional standards, and enhance collaboration between yourself and model validators.\n",
    "\n",
    "<a id='toc2_1_'></a>\n",
    "\n",
    "### Before you begin\n",
    "\n",
    "This notebook assumes you have basic familiarity with Python, including an understanding of how functions work. If you are new to Python, you can still run the notebook but we recommend further familiarizing yourself with the language. \n",
    "\n",
    "If you encounter errors due to missing modules in your Python environment, install the modules with `pip install`, and then re-run the notebook. For more help, refer to [Installing Python Modules](https://docs.python.org/3/installing/index.html).\n",
    "\n",
    "<a id='toc2_2_'></a>\n",
    "\n",
    "### New to ValidMind?\n",
    "\n",
    "If you haven't already seen our documentation on the [ValidMind Library](https://docs.validmind.ai/developer/validmind-library.html), we recommend you begin by exploring the available resources in this section. There, you can learn more about documenting models and running tests, as well as find code samples and our Python Library API reference.\n",
    "\n",
    "<div class=\"alert alert-block alert-info\" style=\"background-color: #B5B5B510; color: black; border: 1px solid #083E44; border-left-width: 5px; box-shadow: 2px 2px 4px rgba(0, 0, 0, 0.2);border-radius: 5px;\"><span style=\"color: #083E44;\"><b>For access to all features available in this notebook, create a free ValidMind account.</b></span>\n",
    "<br></br>\n",
    "Signing up is FREE — <a href=\"https://docs.validmind.ai/guide/configuration/register-with-validmind.html\" style=\"color: #DE257E;\"><b>Register with ValidMind</b></a></div>\n",
    "\n",
    "<a id='toc2_3_'></a>\n",
    "\n",
    "### Key concepts\n",
    "\n",
    "- **FunctionModels**: ValidMind offers support for creating `VMModel` instances from Python functions. This enables us to support any \"model\" by simply using the provided function as the model's `predict` method.\n",
    "- **PipelineModels**: ValidMind models (`VMModel` instances) of any type can be piped together to create a model pipeline. This allows model components to be created and tested/documented independently, and then combined into a single model for end-to-end testing and documentation. We use the `|` operator to pipe models together.\n",
    "- **RAG**: RAG stands for Retrieval Augmented Generation and refers to a wide range of GenAI applications where some form of retrieval is used to add context to the prompt so that the LLM that generates content can refer to it when creating its output. In this notebook, we are going to implement a simple RAG setup using LangChain components.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Prerequisites\n",
    "\n",
    "Let's go ahead and install the `validmind` library if its not already installed... Then we can install the `qdrant-client` library for our vector store and `langchain` for everything else:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install -q validmind"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install -q qdrant-client langchain langchain-openai sentencepiece"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Initialize the ValidMind Library\n",
    "\n",
    "ValidMind generates a unique _code snippet_ for each registered model to connect with your developer environment. You initialize the ValidMind Library with this code snippet, which ensures that your documentation and tests are uploaded to the correct model when you run the notebook.\n",
    "\n",
    "### Get your code snippet\n",
    "\n",
    "1. In a browser, [log in to ValidMind](https://docs.validmind.ai/guide/configuration/log-in-to-validmind.html).\n",
    "\n",
    "2. In the left sidebar, navigate to **Model Inventory** and click **+ Register Model**.\n",
    "\n",
    "3. Enter the model details and click **Continue**. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/register-models-in-inventory.html))\n",
    "\n",
    "   For example, to register a model for use with this notebook, select:\n",
    "\n",
    "   - Documentation template: `Gen AI RAG Template`\n",
    "   - Use case: `Marketing/Sales - Analytics`\n",
    "\n",
    "   You can fill in other options according to your preference.\n",
    "\n",
    "4. Go to **Getting Started** and click **Copy snippet to clipboard**.\n",
    "\n",
    "Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load your model identifier credentials from an `.env` file\n",
    "\n",
    "%load_ext dotenv\n",
    "%dotenv .env\n",
    "\n",
    "# Or replace with your code snippet\n",
    "\n",
    "import validmind as vm\n",
    "\n",
    "vm.init(\n",
    "    # api_host=\"...\",\n",
    "    # api_key=\"...\",\n",
    "    # api_secret=\"...\",\n",
    "    # model=\"...\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Read Open AI API Key\n",
    "\n",
    "We will need to have an OpenAI API key to be able to use their `text-embedding-3-small` model for our embeddings, `gpt-3.5-turbo` model for our generator and `gpt-4o` model for our LLM-as-Judge tests. If you don't have an OpenAI API key, you can get one by signing up at [OpenAI](https://platform.openai.com/signup). Then you can create a `.env` file in the root of your project and the following cell will load it from there. Alternatively, you can just uncomment the line below to directly set the key (not recommended for security reasons)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load openai api key\n",
    "import os\n",
    "\n",
    "import dotenv\n",
    "import nltk\n",
    "\n",
    "dotenv.load_dotenv()\n",
    "nltk.download('stopwords')\n",
    "nltk.download('punkt_tab')\n",
    "\n",
    "# os.environ[\"OPENAI_API_KEY\"] = \"sk-...\"\n",
    "\n",
    "if not \"OPENAI_API_KEY\" in os.environ:\n",
    "    raise ValueError(\"OPENAI_API_KEY is not set\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dataset Loader\n",
    "\n",
    "Great, now that we have all of our dependencies installed, the ValidMind Library initialized and connected to our model and our OpenAI API key setup, we can go ahead and load our datasets. We will use the synthetic `RFP` dataset included with ValidMind for this notebook. This dataset contains a variety of RFP questions and ground truth answers that we can use both as the source where our Retriever will search for similar question-answer pairs as well as our test set for evaluating the performance of our RAG model. To do this, we just have to load it and call the preprocess function to get a split of the data into train and test sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import the sample dataset from the library\n",
    "from validmind.datasets.llm.rag import rfp\n",
    "\n",
    "raw_df = rfp.load_data()\n",
    "train_df, test_df = rfp.preprocess(raw_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vm_train_ds = vm.init_dataset(\n",
    "    train_df,\n",
    "    text_column=\"question\",\n",
    "    target_column=\"ground_truth\",\n",
    ")\n",
    "\n",
    "vm_test_ds = vm.init_dataset(\n",
    "    test_df,\n",
    "    text_column=\"question\",\n",
    "    target_column=\"ground_truth\",\n",
    ")\n",
    "\n",
    "vm_test_ds.df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data validation\n",
    "\n",
    "Now that we have loaded our dataset, we can go ahead and run some data validation tests right away to start assessing and documenting the quality of our data. Since we are using a text dataset, we can use ValidMind's built-in array of text data quality tests to check that things like number of duplicates, missing values, and other common text data issues are not present in our dataset. We can also run some tests to check the sentiment and toxicity of our data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Duplicates\n",
    "\n",
    "First, let's check for duplicates in our dataset. We can use the `validmind.data_validation.Duplicates` test and pass our dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from validmind.tests import run_test\n",
    "\n",
    "run_test(\n",
    "    test_id=\"validmind.data_validation.Duplicates\",\n",
    "    inputs={\"dataset\": vm_train_ds},\n",
    ").log()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Stop Words\n",
    "\n",
    "Next, let's check for stop words in our dataset. We can use the `validmind.data_validation.StopWords` test and pass our dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "run_test(\n",
    "    test_id=\"validmind.data_validation.nlp.StopWords\",\n",
    "    inputs={\n",
    "        \"dataset\": vm_train_ds,\n",
    "    },\n",
    ").log()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Punctuations\n",
    "\n",
    "Next, let's check for punctuations in our dataset. We can use the `validmind.data_validation.Punctuations` test:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "run_test(\n",
    "    test_id=\"validmind.data_validation.nlp.Punctuations\",\n",
    "    inputs={\n",
    "        \"dataset\": vm_train_ds,\n",
    "    },\n",
    ").log()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Common Words\n",
    "\n",
    "Next, let's check for common words in our dataset. We can use the `validmind.data_validation.CommonWord` test:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "run_test(\n",
    "    test_id=\"validmind.data_validation.nlp.CommonWords\",\n",
    "    inputs={\n",
    "        \"dataset\": vm_train_ds,\n",
    "    },\n",
    ").log()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Language Detection\n",
    "\n",
    "For documentation purposes, we can detect and log the languages used in the dataset with the `validmind.data_validation.LanguageDetection` test:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "run_test(\n",
    "    test_id=\"validmind.data_validation.nlp.LanguageDetection\",\n",
    "    inputs={\n",
    "        \"dataset\": vm_train_ds,\n",
    "    },\n",
    ").log()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Toxicity Score\n",
    "\n",
    "Now, let's go ahead and run the `validmind.data_validation.nlp.Toxicity` test to compute a toxicity score for our dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "run_test(\n",
    "    \"validmind.data_validation.nlp.Toxicity\",\n",
    "    inputs={\n",
    "        \"dataset\": vm_train_ds,\n",
    "    },\n",
    ").log()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Polarity and Subjectivity\n",
    "\n",
    "We can also run the `validmind.data_validation.nlp.PolarityAndSubjectivity` test to compute the polarity and subjectivity of our dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "run_test(\n",
    "    \"validmind.data_validation.nlp.PolarityAndSubjectivity\",\n",
    "    inputs={\n",
    "        \"dataset\": vm_train_ds,\n",
    "    },\n",
    ").log()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sentiment\n",
    "\n",
    "Finally, we can run the `validmind.data_validation.nlp.Sentiment` test to plot the sentiment of our dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "run_test(\n",
    "    \"validmind.data_validation.nlp.Sentiment\",\n",
    "    inputs={\n",
    "        \"dataset\": vm_train_ds,\n",
    "    },\n",
    ").log()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Embedding Model\n",
    "\n",
    "Now that we have our dataset loaded and have run some data validation tests to assess and document the quality of our data, we can go ahead and initialize our embedding model. We will use the `text-embedding-3-small` model from OpenAI for this purpose wrapped in the `OpenAIEmbeddings` class from LangChain. This model will be used to \"embed\" our questions both for inserting the question-answer pairs from the \"train\" set into the vector store and for embedding the question from inputs when making predictions with our RAG model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_openai import OpenAIEmbeddings\n",
    "\n",
    "embedding_client = OpenAIEmbeddings(model=\"text-embedding-3-small\")\n",
    "\n",
    "\n",
    "def embed(input):\n",
    "    \"\"\"Returns a text embedding for the given text\"\"\"\n",
    "    return embedding_client.embed_query(input[\"question\"])\n",
    "\n",
    "\n",
    "vm_embedder = vm.init_model(input_id=\"embedding_model\", predict_fn=embed)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What we have done here is to initialize the `OpenAIEmbeddings` class so it uses OpenAI's `text-embedding-3-small` model. We then created an `embed` function that takes in an `input` dictionary and uses the `embed_query` method of the embedding client to compute the embeddings of the `question`. We use an `embed` function since that is how ValidMind supports any custom model. We will use this strategy for the retrieval and generator models as well but you could also use, say, a HuggingFace model directly. See the documentation for more information on which model types are directly supported - [ValidMind Documentation](https://docs.validmind.ai/validmind/validmind.html)... Finally, we use the `init_model` function from the ValidMind Library to create a `VMModel` object that can be used in ValidMind tests. This also logs the model to our model documentation and any test that uses the model will be linked to the logged model and its metadata."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Assign Predictions\n",
    "\n",
    "To precompute the embeddings for our test set, we can call the `assign_predictions` method of our `vm_test_ds` object we created above. This will compute the embeddings for each question in the test set and store them in the a special prediction column of the test set thats linked to our `vm_embedder` model. This will allow us to use these embeddings later when we run tests against our embedding model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vm_test_ds.assign_predictions(vm_embedder)\n",
    "print(vm_test_ds)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Run tests\n",
    "\n",
    "Now that everything is setup for the embedding model, we can go ahead and run some tests to assess and document the quality of our embeddings. We will use the `validmind.model_validation.embeddings.*` tests to compute a variety of metrics against our model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "run_test(\n",
    "    \"validmind.model_validation.embeddings.StabilityAnalysisRandomNoise\",\n",
    "    inputs={\n",
    "        \"model\": vm_embedder,\n",
    "        \"dataset\": vm_test_ds,\n",
    "    },\n",
    "    params={\"probability\": 0.3},\n",
    ").log()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "run_test(\n",
    "    \"validmind.model_validation.embeddings.StabilityAnalysisSynonyms\",\n",
    "    inputs={\n",
    "        \"model\": vm_embedder,\n",
    "        \"dataset\": vm_test_ds,\n",
    "    },\n",
    "    params={\"probability\": 0.3},\n",
    ").log()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "run_test(\n",
    "    \"validmind.model_validation.embeddings.StabilityAnalysisTranslation\",\n",
    "    inputs={\n",
    "        \"model\": vm_embedder,\n",
    "        \"dataset\": vm_test_ds,\n",
    "    },\n",
    "    params={\n",
    "        \"source_lang\": \"en\",\n",
    "        \"target_lang\": \"fr\",\n",
    "    },\n",
    ").log()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "run_test(\n",
    "    \"validmind.model_validation.embeddings.CosineSimilarityHeatmap\",\n",
    "    inputs={\n",
    "        \"model\": vm_embedder,\n",
    "        \"dataset\": vm_test_ds,\n",
    "    },\n",
    ").log()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "run_test(\n",
    "    \"validmind.model_validation.embeddings.CosineSimilarityDistribution\",\n",
    "    inputs={\n",
    "        \"model\": vm_embedder,\n",
    "        \"dataset\": vm_test_ds,\n",
    "    },\n",
    ").log()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "run_test(\n",
    "    \"validmind.model_validation.embeddings.EuclideanDistanceHeatmap\",\n",
    "    inputs={\n",
    "        \"model\": vm_embedder,\n",
    "        \"dataset\": vm_test_ds,\n",
    "    },\n",
    ").log()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "run_test(\n",
    "    \"validmind.model_validation.embeddings.PCAComponentsPairwisePlots\",\n",
    "    inputs={\n",
    "        \"model\": vm_embedder,\n",
    "        \"dataset\": vm_test_ds,\n",
    "    },\n",
    "    params={\"n_components\": 3},\n",
    ").log()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "run_test(\n",
    "    \"validmind.model_validation.embeddings.TSNEComponentsPairwisePlots\",\n",
    "    inputs={\n",
    "        \"model\": vm_embedder,\n",
    "        \"dataset\": vm_test_ds,\n",
    "    },\n",
    "    params={\"n_components\": 3, \"perplexity\": 20},\n",
    ").log()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Setup Vector Store\n",
    "\n",
    "Great, so now that we have assessed our embedding model and verified that it is performing well, we can go ahead and use it to compute embeddings for our question-answer pairs in the \"train\" set. We will then use these embeddings to insert the question-answer pairs into a vector store. We will use an in-memory `qdrant` vector database for demo purposes but any option would work just as well here. We will use the `QdrantClient` class from LangChain to interact with the vector store. This class will allow us to insert and search for embeddings in the vector store."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generate embeddings for the Train Set\n",
    "\n",
    "We can use the same `assign_predictions` method from earlier except this time we will use the `vm_train_ds` object to compute the embeddings for the question-answer pairs in the \"train\" set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vm_train_ds.assign_predictions(vm_embedder)\n",
    "print(vm_train_ds)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Insert embeddings and questions into Vector DB\n",
    "\n",
    "Now that we have computed the embeddings for our question-answer pairs in the \"train\" set, we can go ahead and insert them into the vector store:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_community.vectorstores import Qdrant\n",
    "from langchain_openai import OpenAIEmbeddings\n",
    "from langchain_community.document_loaders import DataFrameLoader\n",
    "\n",
    "# load documents from dataframe\n",
    "loader = DataFrameLoader(train_df, page_content_column=\"question\")\n",
    "docs = loader.load()\n",
    "# choose model using embedding client\n",
    "embedding_client = OpenAIEmbeddings(model=\"text-embedding-3-small\")\n",
    "\n",
    "# setup vector datastore\n",
    "qdrant = Qdrant.from_documents(\n",
    "    docs,\n",
    "    embedding_client,\n",
    "    location=\":memory:\",  # Local mode with in-memory storage only\n",
    "    collection_name=\"rfp_rag_collection\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Retrieval Model\n",
    "\n",
    "Now that we have an embedding model and a vector database setup and loaded with our data, we need a Retrieval model that can search for similar question-answer pairs for a given input question. Once created, we can initialize this as a ValidMind model and `assign_predictions` to it just like our embedding model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "def retrieve(input):\n",
    "    contexts = []\n",
    "\n",
    "    for result in qdrant.similarity_search_with_score(input[\"question\"]):\n",
    "        document, score = result\n",
    "        context = f\"Q: {document.page_content}\\n\"\n",
    "        context += f\"A: {document.metadata['ground_truth']}\\n\"\n",
    "\n",
    "        contexts.append(context)\n",
    "\n",
    "    return contexts\n",
    "\n",
    "\n",
    "vm_retriever = vm.init_model(input_id=\"retrieval_model\", predict_fn=retrieve)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vm_test_ds.assign_predictions(model=vm_retriever)\n",
    "print(vm_test_ds)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Generation Model\n",
    "\n",
    "As the final piece of this simple RAG pipeline, we can create and initialize a generation model that will use the retrieved context to generate an answer to the input question. We will use the `gpt-3.5-turbo` model from OpenAI."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "from openai import OpenAI\n",
    "\n",
    "from validmind.models import Prompt\n",
    "\n",
    "\n",
    "system_prompt = \"\"\"\n",
    "You are an expert RFP AI assistant.\n",
    "You are tasked with answering new RFP questions based on existing RFP questions and answers.\n",
    "You will be provided with the existing RFP questions and answer pairs that are the most relevant to the new RFP question.\n",
    "After that you will be provided with a new RFP question.\n",
    "You will generate an answer and respond only with the answer.\n",
    "Ignore your pre-existing knowledge and answer the question based on the provided context.\n",
    "\"\"\".strip()\n",
    "\n",
    "openai_client = OpenAI()\n",
    "\n",
    "\n",
    "def generate(input):\n",
    "    response = openai_client.chat.completions.create(\n",
    "        model=\"gpt-3.5-turbo\",\n",
    "        messages=[\n",
    "            {\"role\": \"system\", \"content\": system_prompt},\n",
    "            {\"role\": \"user\", \"content\": \"\\n\\n\".join(input[\"retrieval_model\"])},\n",
    "            {\"role\": \"user\", \"content\": input[\"question\"]},\n",
    "        ],\n",
    "    )\n",
    "\n",
    "    return response.choices[0].message.content\n",
    "\n",
    "\n",
    "vm_generator = vm.init_model(\n",
    "    input_id=\"generation_model\",\n",
    "    predict_fn=generate,\n",
    "    prompt=Prompt(template=system_prompt),\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's test it out real quick:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "vm_generator.predict(\n",
    "    pd.DataFrame(\n",
    "        {\"retrieval_model\": [[\"My name is anil\"]], \"question\": [\"what is my name\"]}\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prompt Evaluation\n",
    "\n",
    "Now that we have our generator model initialized, we can run some LLM-as-Judge tests to evaluate the system prompt. This will allow us to get an initial sense of how well the prompt meets a few best practices for prompt engineering. These tests use an LLM to rate the prompt on a scale of 1-10 against the following criteria:\n",
    "\n",
    "- **Examplar Bias**: When using multi-shot prompting, does the prompt contain an unbiased distribution of examples?\n",
    "- **Delimitation**: When using complex prompts containing examples, contextual information, or other elements, is the prompt formatted in such a way that each element is clearly separated?\n",
    "- **Clarity**: How clearly the prompt states the task.\n",
    "- **Conciseness**: How succinctly the prompt states the task.\n",
    "- **Instruction Framing**: Whether the prompt contains negative instructions.\n",
    "- **Specificity**: How specific the prompt defines the task."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "run_test(\n",
    "    \"validmind.prompt_validation.Bias\",\n",
    "    inputs={\n",
    "        \"model\": vm_generator,\n",
    "    },\n",
    ").log()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "run_test(\n",
    "    \"validmind.prompt_validation.Clarity\",\n",
    "    inputs={\n",
    "        \"model\": vm_generator,\n",
    "    },\n",
    ").log()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "run_test(\n",
    "    \"validmind.prompt_validation.Conciseness\",\n",
    "    inputs={\n",
    "        \"model\": vm_generator,\n",
    "    },\n",
    ").log()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "run_test(\n",
    "    \"validmind.prompt_validation.Delimitation\",\n",
    "    inputs={\n",
    "        \"model\": vm_generator,\n",
    "    },\n",
    ").log()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "run_test(\n",
    "    \"validmind.prompt_validation.NegativeInstruction\",\n",
    "    inputs={\n",
    "        \"model\": vm_generator,\n",
    "    },\n",
    ").log()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "run_test(\n",
    "    \"validmind.prompt_validation.Specificity\",\n",
    "    inputs={\n",
    "        \"model\": vm_generator,\n",
    "    },\n",
    ").log()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Setup RAG Pipeline Model\n",
    "\n",
    "Now that we have all of our individual \"component\" models setup and initialized we need some way to put them all together in a single \"pipeline\". We can use the `PipelineModel` class to do this. This ValidMind model type simply wraps any number of other ValidMind models and runs them in sequence. We can use a pipe(`|`) operator - in Python this is normally an `or` operator but we have overloaded it for easy pipeline creation - to chain together our models. We can then initialize this pipeline model and assign predictions to it just like any other model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "vm_rag_model = vm.init_model(vm_retriever | vm_generator, input_id=\"rag_model\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can `assign_predictions` to the pipeline model just like we did with the individual models. This will run the pipeline on the test set and store the results in the test set for later use."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vm_test_ds.assign_predictions(model=vm_rag_model)\n",
    "print(vm_test_ds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vm_test_ds._df.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Run tests\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## RAGAS evaluation\n",
    "\n",
    "Let's go ahead and run some of our new RAG tests against our model...\n",
    "\n",
    "> Note: these tests are still being developed and are not yet in a stable state. We are using advanced tests here that use LLM-as-Judge and other strategies to assess things like the relevancy of the retrieved context to the input question and the correctness of the generated answer when compared to the ground truth. There is more to come in this area so stay tuned!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings\n",
    "\n",
    "warnings.filterwarnings(\"ignore\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Semantic Similarity\n",
    "\n",
    "The concept of Answer Semantic Similarity pertains to the assessment of the semantic resemblance between the generated answer and the ground truth. This evaluation is based on the ground truth and the answer, with values falling within the range of 0 to 1. A higher score signifies a better alignment between the generated answer and the ground truth.\n",
    "\n",
    "Measuring the semantic similarity between answers can offer valuable insights into the quality of the generated response. This evaluation utilizes a cross-encoder model to calculate the semantic similarity score."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "run_test(\n",
    "    \"validmind.model_validation.ragas.SemanticSimilarity\",\n",
    "    inputs={\"dataset\": vm_test_ds},\n",
    "    params={\n",
    "        \"response_column\": \"rag_model_prediction\",\n",
    "        \"reference_column\": \"ground_truth\",\n",
    "    },\n",
    ").log()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Context Entity Recall\n",
    "\n",
    "This test gives the measure of recall of the retrieved context, based on the number of entities present in both ground_truths and contexts relative to the number of entities present in the ground_truths alone. Simply put, it is a measure of what fraction of entities are recalled from ground_truths. This test is useful in fact-based use cases like tourism help desk, historical QA, etc. This test can help evaluate the retrieval mechanism for entities, based on comparison with entities present in ground_truths, because in cases where entities matter, we need the contexts which cover them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "run_test(\n",
    "    \"validmind.model_validation.ragas.ContextEntityRecall\",\n",
    "    inputs={\"dataset\": vm_test_ds},\n",
    "    params={\n",
    "        \"reference_column\": \"ground_truth\",\n",
    "        \"retrieved_contexts_column\": \"retrieval_model_prediction\",\n",
    "    },\n",
    ").log()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Context Precision\n",
    "\n",
    "Context Precision is a test that evaluates whether all of the ground-truth relevant items present in the contexts are ranked higher or not. Ideally all the relevant chunks must appear at the top ranks. This test is computed using the question, ground_truth and the contexts, with values ranging between 0 and 1, where higher scores indicate better precision."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "run_test(\n",
    "    \"validmind.model_validation.ragas.ContextPrecision\",\n",
    "    inputs={\"dataset\": vm_test_ds},\n",
    "    params={\n",
    "        \"user_input_column\": \"question\",\n",
    "        \"retrieved_contexts_column\": \"retrieval_model_prediction\",\n",
    "        \"reference_column\": \"ground_truth\",\n",
    "    },\n",
    ").log()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Context Precision Without Reference\n",
    "\n",
    "This test evaluates whether retrieved  contexts align well with the expected response for a given user input, without requiring a ground-truth reference. This test assesses the relevance of each retrieved context chunk by comparing it directly to the response."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "run_test(\n",
    "    \"validmind.model_validation.ragas.ContextPrecisionWithoutReference\",\n",
    "    inputs={\"dataset\": vm_test_ds},\n",
    "    params={\n",
    "        \"user_input_column\": \"question\",\n",
    "        \"retrieved_contexts_column\": \"retrieval_model_prediction\",\n",
    "        \"response_column\": \"rag_model_prediction\",\n",
    "    },\n",
    ").log()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Faithfulness\n",
    "\n",
    "This measures the factual consistency of the generated answer against the given context. It is calculated from answer and retrieved context. The answer is scaled to (0,1) range. Higher the better.\n",
    "\n",
    "The generated answer is regarded as faithful if all the claims that are made in the answer can be inferred from the given context. To calculate this a set of claims from the generated answer is first identified. Then each one of these claims are cross checked with given context to determine if it can be inferred from given context or not."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "run_test(\n",
    "    \"validmind.model_validation.ragas.Faithfulness\",\n",
    "    inputs={\"dataset\": vm_test_ds},\n",
    "    params={\n",
    "        \"user_input_column\": \"question\",\n",
    "        \"response_column\": \"rag_model_prediction\",\n",
    "        \"retrieved_contexts_column\": \"retrieval_model_prediction\",\n",
    "    },\n",
    ").log()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Response Relevancy\n",
    "\n",
    "The Response Relevancy test, focuses on assessing how pertinent the generated answer is to the given prompt. A lower score is assigned to answers that are incomplete or contain redundant information and higher scores indicate better relevancy. This test is computed using the question, the context and the answer.\n",
    "\n",
    "The Response Relevancy is defined as the mean cosine similartiy of the original question to a number of artifical questions, which where generated (reverse engineered) based on the answer.\n",
    "\n",
    "Please note, that eventhough in practice the score will range between 0 and 1 most of the time, this is not mathematically guranteed, due to the nature of the cosine similarity ranging from -1 to 1.\n",
    "\n",
    "> Note: This is a reference free test. If you’re looking to compare ground truth answer with generated answer refer to Answer Correctness.\n",
    "\n",
    "An answer is deemed relevant when it directly and appropriately addresses the original question. Importantly, our assessment of answer relevance does not consider factuality but instead penalizes cases where the answer lacks completeness or contains redundant details. To calculate this score, the LLM is prompted to generate an appropriate question for the generated answer multiple times, and the mean cosine similarity between these generated questions and the original question is measured. The underlying idea is that if the generated answer accurately addresses the initial question, the LLM should be able to generate questions from the answer that align with the original question."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "run_test(\n",
    "    \"validmind.model_validation.ragas.ResponseRelevancy\",\n",
    "    inputs={\"dataset\": vm_test_ds},\n",
    "    params={\n",
    "        \"user_input_column\": \"question\",\n",
    "        \"response_column\": \"rag_model_prediction\",\n",
    "        \"retrieved_contexts_column\": \"retrieval_model_prediction\",\n",
    "    },\n",
    ").log()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Context Recall\n",
    "\n",
    "Context recall measures the extent to which the retrieved context aligns with the annotated answer, treated as the ground truth. It is computed based on the ground truth and the retrieved context, and the values range between 0 and 1, with higher values indicating better performance.\n",
    "\n",
    "To estimate context recall from the ground truth answer, each sentence in the ground truth answer is analyzed to determine whether it can be attributed to the retrieved context or not. In an ideal scenario, all sentences in the ground truth answer should be attributable to the retrieved context."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "run_test(\n",
    "    \"validmind.model_validation.ragas.ContextRecall\",\n",
    "    inputs={\"dataset\": vm_test_ds},\n",
    "    params={\n",
    "        \"user_input_column\": \"question\",\n",
    "        \"retrieved_contexts_column\": \"retrieval_model_prediction\",\n",
    "        \"reference_column\": \"ground_truth\",\n",
    "    },\n",
    ").log()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Answer Correctness\n",
    "\n",
    "The assessment of Answer Correctness involves gauging the accuracy of the generated answer when compared to the ground truth. This evaluation relies on the ground truth and the answer, with scores ranging from 0 to 1. A higher score indicates a closer alignment between the generated answer and the ground truth, signifying better correctness.\n",
    "\n",
    "Answer correctness encompasses two critical aspects: semantic similarity between the generated answer and the ground truth, as well as factual similarity. These aspects are combined using a weighted scheme to formulate the answer correctness score.\n",
    "\n",
    "Factual correctness quantifies the factual overlap between the generated answer and the ground truth answer. This is done using the concepts of:\n",
    "\n",
    "- TP (True Positive): Facts or statements that are present in both the ground truth and the generated answer.\n",
    "- FP (False Positive): Facts or statements that are present in the generated answer but not in the ground truth.\n",
    "- FN (False Negative): Facts or statements that are present in the ground truth but not in the generated answer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "run_test(\n",
    "    \"validmind.model_validation.ragas.AnswerCorrectness\",\n",
    "    inputs={\"dataset\": vm_test_ds},\n",
    "    params={\n",
    "        \"user_input_column\": \"question\",\n",
    "        \"response_column\": \"rag_model_prediction\",\n",
    "        \"reference_column\": \"ground_truth\",\n",
    "    },\n",
    ").log()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Aspect Critic\n",
    "\n",
    "This is designed to assess submissions based on predefined aspects such as harmlessness and correctness. Additionally, users have the flexibility to define their own aspects for evaluating submissions according to their specific criteria. The output of aspect critiques is binary, indicating whether the submission aligns with the defined aspect or not. This evaluation is performed using the ‘answer’ as input.\n",
    "\n",
    "Critiques within the LLM evaluators evaluate submissions based on the provided aspect. Ragas Critiques offers a range of predefined aspects like correctness, harmfulness, etc. Users can also define their own aspects for evaluating submissions based on their specific criteria. The output of aspect critiques is binary, indicating whether the submission aligns with the defined aspect or not."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "run_test(\n",
    "    \"validmind.model_validation.ragas.AspectCritic\",\n",
    "    inputs={\"dataset\": vm_test_ds},\n",
    "    params={\n",
    "        \"user_input_column\": \"question\",\n",
    "        \"response_column\": \"rag_model_prediction\",\n",
    "        \"retrieved_contexts_column\": \"retrieval_model_prediction\",\n",
    "    },\n",
    ").log()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Noise Sensitivity\n",
    "\n",
    "This test is designed to evaluate the robustness of the RAG pipeline model against noise in the retrieved context. It works by checking how well the \"claims\" in the generated answer match up with the \"claims\" in the ground truth answer. If the generated answer contains \"claims\" from the contexts that the ground truth answer does not contain, those claims are considered incorrect. The score for each answer is the number of incorrect claims divided by the total number of claims. This *can* be interpreted as a measure of how sensitive the LLM is to \"noise\" in the context where \"noise\" is information that is relevant but should not be included in the answer since the ground truth answer does not contain it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "run_test(\n",
    "    \"validmind.model_validation.ragas.NoiseSensitivity\",\n",
    "    inputs={\"dataset\": vm_test_ds},\n",
    "    params={\n",
    "        \"user_input_column\": \"question\",\n",
    "        \"response_column\": \"rag_model_prediction\",\n",
    "        \"reference_column\": \"ground_truth\",\n",
    "        \"retrieved_contexts_column\": \"retrieval_model_prediction\",\n",
    "    },\n",
    ").log()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generation quality\n",
    "\n",
    "In this section, we evaluate the alignment and relevance of generated responses to reference outputs within our retrieval-augmented generation (RAG) application. We use metrics that assess various quality dimensions of the generated responses, including semantic similarity, structural alignment, and phrasing overlap. Semantic similarity metrics compare embeddings of generated and reference text to capture deeper contextual alignment, while overlap and alignment measures quantify how well the phrasing and structure of generated responses match the intended outputs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Token Disparity\n",
    "\n",
    "This test assesses the difference in token counts between the reference texts (ground truth) and the answers generated by the RAG model. It helps evaluate how well the model's outputs align with the expected length and level of detail in the reference texts. A significant disparity in token counts could signal issues with generation quality, such as excessive verbosity or insufficient detail. Consistently low token counts in generated answers compared to references might suggest that the model’s outputs are incomplete or overly concise, missing important contextual information."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "run_test(\n",
    "    \"validmind.model_validation.TokenDisparity\",\n",
    "    inputs={\n",
    "        \"dataset\": vm_test_ds,\n",
    "        \"model\": vm_rag_model,\n",
    "    },\n",
    ").log()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ROUGE Score\n",
    "\n",
    "This test evaluates the quality of answers generated by the RAG model by measuring overlaps in n-grams, word sequences, and word pairs between the model output and the reference (ground truth) text. ROUGE, short for Recall-Oriented Understudy for Gisting Evaluation, assesses both precision and recall, providing a balanced view of how well the generated response captures the reference content. ROUGE precision measures the proportion of n-grams in the generated text that match the reference, highlighting relevance and conciseness, while ROUGE recall assesses the proportion of reference n-grams present in the generated text, indicating completeness and thoroughness. \n",
    "\n",
    "Low precision scores might reveal that the generated text includes redundant or irrelevant information, while low recall scores suggest omissions of essential details from the reference. Consistently low ROUGE scores could indicate poor overall alignment with the ground truth, suggesting the model may be missing key content or failing to capture the intended meaning."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "run_test(\n",
    "    \"validmind.model_validation.RougeScore\",\n",
    "    inputs={\n",
    "        \"dataset\": vm_test_ds,\n",
    "        \"model\": vm_rag_model,\n",
    "    },\n",
    "    params={\n",
    "        \"metric\": \"rouge-1\",\n",
    "    },\n",
    ").log()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### BLEU Score\n",
    "\n",
    "The BLEU Score test evaluates the quality of answers generated by the RAG model by measuring n-gram overlap between the generated text and the reference (ground truth) text, with a specific focus on exact precision in phrasing. While ROUGE precision also assesses overlap, BLEU differs in two main ways: first, it applies a geometric average across multiple n-gram levels, capturing precise phrase alignment, and second, it includes a brevity penalty to prevent overly short outputs from inflating scores artificially. This added precision focus is valuable in RAG applications where strict adherence to reference language is essential, as BLEU emphasizes the match to exact phrasing. In contrast, ROUGE precision evaluates general content overlap without penalizing brevity, offering a broader sense of content alignment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "run_test(\n",
    "    \"validmind.model_validation.BleuScore\",\n",
    "    inputs={\n",
    "        \"dataset\": vm_test_ds,\n",
    "        \"model\": vm_rag_model,\n",
    "    },\n",
    ").log()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### BERT Score\n",
    "\n",
    "This test evaluates the quality of the RAG generated answers using BERT embeddings to measure precision, recall, and F1 scores based on semantic similarity, rather than exact n-gram matches as in BLEU and ROUGE. This approach captures contextual meaning, making it valuable when wording differs but the intended message closely aligns with the reference. In RAG applications, the BERT score is especially useful for ensuring that generated answers convey the reference text’s meaning, even if phrasing varies. Consistently low scores indicate a lack of semantic alignment, suggesting the model may miss or misrepresent key content. Low precision may reflect irrelevant or redundant details, while low recall can indicate omissions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "run_test(\n",
    "    \"validmind.model_validation.BertScore\",\n",
    "    inputs={\n",
    "        \"dataset\": vm_test_ds,\n",
    "        \"model\": vm_rag_model,\n",
    "    },\n",
    ").log()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### METEOR Score\n",
    "\n",
    "This test evaluates the quality of the generated answers by measuring alignment with the ground truth, emphasizing both accuracy and fluency. Unlike BLEU and ROUGE, which focus on n-gram matches, METEOR combines precision, recall, synonym matching, and word order, focusing at how well the generated text conveys meaning and reads naturally. This metric is especially useful for RAG applications where sentence structure and natural flow are crucial for clear communication. Lower scores may suggest alignment issues, indicating that the answers may lack fluency or key content. Discrepancies in word order or high fragmentation penalties can reveal problems with how the model constructs sentences, potentially affecting readability."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "run_test(\n",
    "    \"validmind.model_validation.MeteorScore\",\n",
    "    inputs={\n",
    "        \"dataset\": vm_test_ds,\n",
    "        \"model\": vm_rag_model,\n",
    "    },\n",
    ").log()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bias and Toxicity\n",
    "\n",
    "In this section, we use metrics like Toxicity Score and Regard Score to evaluate both the generated responses and the ground truth. These tests helps us detect any harmful, offensive, or inappropriate language and evaluate the level of bias and neutrality enabling us to assess and mitigate potential biases in both the model's responses and the original dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Toxicity Score\n",
    "\n",
    "This test measures the level of harmful or offensive content in the generated answers. The test uses a preloaded toxicity detection tool from Hugging Face, which identifies language that may be inappropriate, aggressive, or derogatory. High toxicity scores indicate potentially toxic content, while consistently elevated scores across multiple outputs may signal underlying issues in the model’s generation process that require attention to prevent the spread of harmful language."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "run_test(\n",
    "    \"validmind.model_validation.ToxicityScore\",\n",
    "    inputs={\n",
    "        \"dataset\": vm_test_ds,\n",
    "        \"model\": vm_rag_model,\n",
    "    },\n",
    ").log()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Regard Score\n",
    "\n",
    "This test evaluates the sentiment and perceived regard—categorized as positive, negative, neutral, or other—in answers generated by the RAG model. This is important for identifying any biases or sentiment tendencies in responses, ensuring that generated answers are balanced and appropriate for the context. The uses a preloaded regard evaluation tool from Hugging Face to compute scores for each response. High skewness in regard scores, especially if the generated responses consistently diverge from expected sentiments in the reference texts, may reveal biases in the model’s generation, such as overly positive or negative tones where neutrality is expected."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "run_test(\n",
    "    \"validmind.model_validation.RegardScore\",\n",
    "    inputs={\n",
    "        \"dataset\": vm_test_ds,\n",
    "        \"model\": vm_rag_model,\n",
    "    },\n",
    ").log()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Conclusion\n",
    "\n",
    "In this notebook, we have seen how we can use LangChain and ValidMind together to build, evaluate and document a simple RAG Model as its developed. This is a great example of the interactive development experience that ValidMind is designed to support. We can quickly iterate on our model and document as we go... We have seen how ValidMind supports non-traditional \"models\" using a functional interface and how we can build pipelines of many models to support complex GenAI workflows.\n",
    "\n",
    "This is still a work in progress and we are actively developing new tests to support more advanced GenAI workflows. We are also keeping an eye on the most popular GenAI models and libraries to explore direct integrations. Stay tuned for more updates and new features in this area!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Upgrade ValidMind\n",
    "\n",
    "<div class=\"alert alert-block alert-info\" style=\"background-color: #B5B5B510; color: black; border: 1px solid #083E44; border-left-width: 5px; box-shadow: 2px 2px 4px rgba(0, 0, 0, 0.2);border-radius: 5px;\">After installing ValidMind, you’ll want to periodically make sure you are on the latest version to access any new features and other enhancements.</div>\n",
    "\n",
    "Retrieve the information for the currently installed version of ValidMind:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip show validmind"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If the version returned is lower than the version indicated in our [production open-source code](https://github.com/validmind/validmind-library/blob/prod/validmind/__version__.py), restart your notebook and run:\n",
    "\n",
    "```bash\n",
    "%pip install --upgrade validmind\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You may need to restart your kernel after running the upgrade package for changes to be applied."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "validmind-py3.10",
   "language": "python",
   "name": "validmind-py3.10"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
