{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Summarization of financial data using a large language model (LLM)\n",
    "\n",
    "Document a large language model (LLM) using the ValidMind Library. The use case is a summarization of financial news based on a dataset containing just over 300k unique news articles as written by journalists at CNN and the Daily Mail. \n",
    "\n",
    "This interactive notebook shows you how to set up the ValidMind Library, initialize the library, and load the dataset, followed by running the model validation tests provided by the ValidMind Library to quickly generate documentation about the data and model.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Before you begin\n",
    "\n",
    "<div class=\"alert alert-block alert-info\" style=\"background-color: #B5B5B510; color: black; border: 1px solid #083E44; border-left-width: 5px; box-shadow: 2px 2px 4px rgba(0, 0, 0, 0.2);border-radius: 5px;\"><span style=\"color: #083E44;\"><b>For access to all features available in this notebook, create a free ValidMind account.</b></span>\n",
    "<br></br>\n",
    "Signing up is FREE — <a href=\"https://docs.validmind.ai/guide/configuration/register-with-validmind.html\" style=\"color: #DE257E;\"><b>Register with ValidMind</b></a></div>\n",
    "\n",
    "This notebook requires an OpenAI API secret key to run. If you don't have one, visit [API keys](https://platform.openai.com/account/api-keys) on OpenAI's site to create a new key for yourself. Note that API usage charges may apply.\n",
    "\n",
    "If you encounter errors due to missing modules in your Python environment, install the modules with `pip install`, and then re-run the notebook. For more help, refer to [Installing Python Modules](https://docs.python.org/3/installing/index.html).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## About ValidMind\n",
    "\n",
    "ValidMind's suite of tools enables organizations to identify, document, and manage model risks for all types of models, including AI/ML models, LLMs, and statistical models. As a model developer, you use the ValidMind Library to automate documentation and validation tests, and then use the ValidMind Platform to collaborate on model documentation, these products simplify model risk management, facilitate compliance with regulations and institutional standards, and enhance collaboration between yourself and model validators.\n",
    "\n",
    "If this is your first time trying out ValidMind, we recommend going through the following resources first:\n",
    "\n",
    "- [Get started](https://docs.validmind.ai/get-started/get-started.html) — The basics, including key concepts, and how our products work\n",
    "- [ValidMind Library](https://docs.validmind.ai/developer/validmind-library.html) — The path for developers, more code samples, and our developer reference\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Install the ValidMind Library\n",
    "\n",
    "To install the library:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install -q validmind"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Initialize the ValidMind Library\n",
    "\n",
    "ValidMind generates a unique _code snippet_ for each registered model to connect with your developer environment. You initialize the ValidMind Library with this code snippet, which ensures that your documentation and tests are uploaded to the correct model when you run the notebook.\n",
    "\n",
    "### Get your code snippet\n",
    "\n",
    "1. In a browser, [log in to ValidMind](https://docs.validmind.ai/guide/configuration/log-in-to-validmind.html).\n",
    "\n",
    "2. In the left sidebar, navigate to **Model Inventory** and click **+ Register Model**.\n",
    "\n",
    "3. Enter the model details and click **Continue**. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/register-models-in-inventory.html))\n",
    "\n",
    "   For example, to register a model for use with this notebook, select:\n",
    "\n",
    "   - Documentation template: `LLM-based Text Summarization`\n",
    "\n",
    "   You can fill in other options according to your preference.\n",
    "\n",
    "4. Go to **Getting Started** and click **Copy snippet to clipboard**.\n",
    "\n",
    "Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load your model identifier credentials from an `.env` file\n",
    "\n",
    "%load_ext dotenv\n",
    "%dotenv .env\n",
    "\n",
    "# Or replace with your code snippet\n",
    "\n",
    "import validmind as vm\n",
    "\n",
    "vm.init(\n",
    "    # api_host=\"...\",\n",
    "    # api_key=\"...\",\n",
    "    # api_secret=\"...\",\n",
    "    # model=\"...\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preview the documentation template\n",
    "\n",
    "A template predefines sections for your model documentation and provides a general outline to follow, making the documentation process much easier.\n",
    "\n",
    "You will upload documentation and test results into this template later on. For now, take a look at the structure that the template provides with the `vm.preview_template()` function from the ValidMind library and note the empty sections:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vm.preview_template()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Helper functions\n",
    "\n",
    "Let's define the following functions to help visualize datasets with long text fields:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import textwrap\n",
    "\n",
    "from IPython.display import display, HTML\n",
    "from tabulate import tabulate\n",
    "\n",
    "\n",
    "def _format_cell_text(text, width=50):\n",
    "    \"\"\"Private function to format a cell's text.\"\"\"\n",
    "    return \"\\n\".join([textwrap.fill(line, width=width) for line in text.split(\"\\n\")])\n",
    "\n",
    "\n",
    "def _format_dataframe_for_tabulate(df):\n",
    "    \"\"\"Private function to format the entire DataFrame for tabulation.\"\"\"\n",
    "    df_out = df.copy()\n",
    "\n",
    "    # Format all string columns\n",
    "    for column in df_out.columns:\n",
    "        # Check if column is of type object (likely strings)\n",
    "        if df_out[column].dtype == object:\n",
    "            df_out[column] = df_out[column].apply(_format_cell_text)\n",
    "    return df_out\n",
    "\n",
    "\n",
    "def _dataframe_to_html_table(df):\n",
    "    \"\"\"Private function to convert a DataFrame to an HTML table.\"\"\"\n",
    "    headers = df.columns.tolist()\n",
    "    table_data = df.values.tolist()\n",
    "    return tabulate(table_data, headers=headers, tablefmt=\"html\")\n",
    "\n",
    "\n",
    "def display_formatted_dataframe(df, num_rows=None):\n",
    "    \"\"\"Primary function to format and display a DataFrame.\"\"\"\n",
    "    if num_rows is not None:\n",
    "        df = df.head(num_rows)\n",
    "    formatted_df = _format_dataframe_for_tabulate(df)\n",
    "    html_table = _dataframe_to_html_table(formatted_df)\n",
    "    display(HTML(html_table))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load the dataset\n",
    "\n",
    "The CNN Dailymail Dataset is an English-language dataset containing just over 300k unique news articles as written by journalists at CNN and the Daily Mail (https://huggingface.co/datasets/cnn_dailymail). The current version supports both extractive and abstractive summarization, though the original version was created for machine reading and comprehension and abstractive question answering.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "df = pd.read_csv(\"./datasets/cnn_dailymail_100_with_predictions.csv\")\n",
    "display_formatted_dataframe(df, num_rows=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get ready to run the analysis\n",
    "\n",
    "Import the ValidMind `FoundationModel` and `Prompt` classes needed for the sentiment analysis later on:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from validmind.models import FoundationModel, Prompt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check your access to the OpenAI API:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "import dotenv\n",
    "import nltk\n",
    "\n",
    "dotenv.load_dotenv()\n",
    "nltk.download('stopwords')\n",
    "\n",
    "if os.getenv(\"OPENAI_API_KEY\") is None:\n",
    "    raise Exception(\"OPENAI_API_KEY not found\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from openai import OpenAI\n",
    "\n",
    "model = OpenAI()\n",
    "\n",
    "\n",
    "def call_model(prompt):\n",
    "    return (\n",
    "        model.chat.completions.create(\n",
    "            model=\"gpt-3.5-turbo\",\n",
    "            messages=[\n",
    "                {\"role\": \"user\", \"content\": prompt},\n",
    "            ],\n",
    "        )\n",
    "        .choices[0]\n",
    "        .message.content\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set the prompt guidelines for the sentiment analysis:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "prompt_template = \"\"\"\n",
    "You are an AI with expertise in summarizing financial news.\n",
    "Your task is to provide a concise summary of the specific news article provided below.\n",
    "Before proceeding, take a moment to understand the context and nuances of the financial terminology used in the article.\n",
    "\n",
    "Article to Summarize:\n",
    "\n",
    "```\n",
    "{article}\n",
    "```\n",
    "\n",
    "Please respond with a concise summary of the article's main points.\n",
    "Ensure that your summary is based on the content of the article and not on external information or assumptions.\n",
    "\"\"\".strip()\n",
    "\n",
    "prompt_variables = [\"article\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vm_test_ds = vm.init_dataset(\n",
    "    dataset=df,\n",
    "    input_id=\"test_dataset\",\n",
    "    text_column=\"article\",\n",
    "    target_column=\"highlights\",\n",
    ")\n",
    "\n",
    "vm_model = vm.init_model(\n",
    "    model=FoundationModel(\n",
    "        predict_fn=call_model,\n",
    "        prompt=Prompt(\n",
    "            template=prompt_template,\n",
    "            variables=prompt_variables,\n",
    "        ),\n",
    "    ),\n",
    "    input_id=\"gpt_35_model\",\n",
    ")\n",
    "\n",
    "# Assign model predictions to the test dataset\n",
    "vm_test_ds.assign_predictions(vm_model, prediction_column=\"gpt_35_prediction\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Run model validation tests\n",
    "\n",
    "It's possible to run a subset of tests on the documentation template by passing a `section` parameter to `run_documentation_tests()`. Let's run the tests that evaluate the model's overall performance (including summarization metrics), by selecting the \"model development\" section of the template:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "summarization_results = vm.run_documentation_tests(\n",
    "    section=\"model_development\",\n",
    "    inputs={\n",
    "        \"dataset\": vm_test_ds,\n",
    "        \"model\": vm_model,\n",
    "    },\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Next steps\n",
    "\n",
    "You can look at the results of this test suite right in the notebook where you ran the code, as you would expect. But there is a better way: view the prompt validation test results as part of your model documentation in the ValidMind Platform:\n",
    "\n",
    "1. In the ValidMind Platform, go to the **Documentation** page for the model you registered earlier. ([Need more help?](https://docs.validmind.ai/guide/model-documentation/working-with-model-documentation.html)\n",
    "\n",
    "2. Expand **2. Data Preparation** or **3. Model Development** to review all test results.\n",
    "\n",
    "What you can see now is a more easily consumable version of the prompt validation testing you just performed, along with other parts of your model documentation that still need to be completed.\n",
    "\n",
    "If you want to learn more about where you are in the model documentation process, take a look our documentation on the [ValidMind Library](https://docs.validmind.ai/developer/validmind-library.html).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Upgrade ValidMind\n",
    "\n",
    "<div class=\"alert alert-block alert-info\" style=\"background-color: #B5B5B510; color: black; border: 1px solid #083E44; border-left-width: 5px; box-shadow: 2px 2px 4px rgba(0, 0, 0, 0.2);border-radius: 5px;\">After installing ValidMind, you’ll want to periodically make sure you are on the latest version to access any new features and other enhancements.</div>\n",
    "\n",
    "Retrieve the information for the currently installed version of ValidMind:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip show validmind"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If the version returned is lower than the version indicated in our [production open-source code](https://github.com/validmind/validmind-library/blob/prod/validmind/__version__.py), restart your notebook and run:\n",
    "\n",
    "```bash\n",
    "%pip install --upgrade validmind\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You may need to restart your kernel after running the upgrade package for changes to be applied."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
