{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sentiment analysis of financial data using a large language model (LLM)\n",
    "\n",
    "Document a large language model (LLM) specialized in sentiment analysis for financial news using the ValidMind Library. \n",
    "\n",
    "This interactive notebook shows you how to set up the ValidMind Library, initializes the library, and uses a specific prompt template for analyzing the sentiment of sentences in a dataset. The notebook also includes example data to test the model's ability to correctly identify sentiment as positive, negative, or neutral.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## About ValidMind\n",
    "\n",
    "ValidMind's suite of tools enables organizations to identify, document, and manage model risks for all types of models, including AI/ML models, LLMs, and statistical models. As a model developer, you use the ValidMind Library to automate documentation and validation tests, and then use the ValidMind Platform to collaborate on documentation initiatives. Together, these products simplify model risk management, facilitate compliance with regulations and institutional standards, and enhance collaboration between yourself and model validators.\n",
    "\n",
    "If this is your first time trying out ValidMind, we recommend going through the following resources first:\n",
    "\n",
    "- [Get started](https://docs.validmind.ai/get-started/get-started.html) — The basics, including key concepts, and how our products work\n",
    "- [ValidMind Library](https://docs.validmind.ai/developer/validmind-library.html) — The path for developers, more code samples, and our developer reference\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Before you begin\n",
    "\n",
    "<div class=\"alert alert-block alert-info\" style=\"background-color: #B5B5B510; color: black; border: 1px solid #083E44; border-left-width: 5px; box-shadow: 2px 2px 4px rgba(0, 0, 0, 0.2);border-radius: 5px;\"><span style=\"color: #083E44;\"><b>For access to all features available in this notebook, create a free ValidMind account.</b></span>\n",
    "<br></br>\n",
    "Signing up is FREE — <a href=\"https://docs.validmind.ai/guide/configuration/register-with-validmind.html\" style=\"color: #DE257E;\"><b>Register with ValidMind</b></a></div>\n",
    "\n",
    "This notebook requires an OpenAI API secret key to run. If you don't have one, visit [API keys](https://platform.openai.com/account/api-keys) on OpenAI's site to create a new key for yourself. Note that API usage charges may apply.\n",
    "\n",
    "If you encounter errors due to missing modules in your Python environment, install the modules with `pip install`, and then re-run the notebook. For more help, refer to [Installing Python Modules](https://docs.python.org/3/installing/index.html).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Install the ValidMind Library\n",
    "\n",
    "To install the library:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install -q validmind"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Initialize the ValidMind Library\n",
    "\n",
    "ValidMind generates a unique _code snippet_ for each registered model to connect with your developer environment. You initialize the ValidMind Library with this code snippet, which ensures that your documentation and tests are uploaded to the correct model when you run the notebook.\n",
    "\n",
    "### Get your code snippet\n",
    "\n",
    "1. In a browser, [log in to ValidMind](https://docs.validmind.ai/guide/configuration/log-in-to-validmind.html).\n",
    "\n",
    "2. In the left sidebar, navigate to **Model Inventory** and click **+ Register Model**.\n",
    "\n",
    "3. Enter the model details and click **Continue**. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/register-models-in-inventory.html))\n",
    "\n",
    "   For example, to register a model for use with this notebook, select:\n",
    "\n",
    "   - Documentation template: `LLM-based Text Classification`\n",
    "   - Use case: `Marketing/Sales - Analytics`\n",
    "\n",
    "   You can fill in other options according to your preference.\n",
    "\n",
    "4. Go to **Getting Started** and click **Copy snippet to clipboard**.\n",
    "\n",
    "Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load your model identifier credentials from an `.env` file\n",
    "\n",
    "%load_ext dotenv\n",
    "%dotenv .env\n",
    "\n",
    "# Or replace with your code snippet\n",
    "\n",
    "import validmind as vm\n",
    "\n",
    "vm.init(\n",
    "    # api_host=\"...\",\n",
    "    # api_key=\"...\",\n",
    "    # api_secret=\"...\",\n",
    "    # model=\"...\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preview the documentation template\n",
    "\n",
    "A template predefines sections for your model documentation and provides a general outline to follow, making the documentation process much easier.\n",
    "\n",
    "You will upload documentation and test results into this template later on. For now, take a look at the structure that the template provides with the `vm.preview_template()` function from the ValidMind library and note the empty sections:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vm.preview_template()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get ready to run the analysis\n",
    "\n",
    "Import the ValidMind `FoundationModel` and `Prompt` classes needed for the sentiment analysis later on:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from validmind.models import FoundationModel, Prompt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check your access to the OpenAI API:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "import dotenv\n",
    "\n",
    "dotenv.load_dotenv()\n",
    "\n",
    "if os.getenv(\"OPENAI_API_KEY\") is None:\n",
    "    raise Exception(\"OPENAI_API_KEY not found\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from openai import OpenAI\n",
    "\n",
    "model = OpenAI()\n",
    "\n",
    "\n",
    "def call_model(prompt):\n",
    "    return (\n",
    "        model.chat.completions.create(\n",
    "            model=\"gpt-3.5-turbo\",\n",
    "            messages=[\n",
    "                {\"role\": \"user\", \"content\": prompt},\n",
    "            ],\n",
    "        )\n",
    "        .choices[0]\n",
    "        .message.content\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set the prompt guidelines for the sentiment analysis:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "prompt_template = \"\"\"\n",
    "You are an AI with expertise in sentiment analysis, particularly in the context of financial news.\n",
    "Your task is to analyze the sentiment of a specific sentence provided below.\n",
    "Before proceeding, take a moment to understand the context and nuances of the financial terminology used in the sentence.\n",
    "\n",
    "Sentence to Analyze:\n",
    "```\n",
    "{Sentence}\n",
    "```\n",
    "\n",
    "Please respond with the sentiment of the sentence denoted by one of either 'positive', 'negative', or 'neutral'.\n",
    "Please respond only with the sentiment enum value. Do not include any other text in your response.\n",
    "\n",
    "Note: Ensure that your analysis is based on the content of the sentence and not on external information or assumptions.\n",
    "\"\"\".strip()\n",
    "\n",
    "prompt_variables = [\"Sentence\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get your sample dataset ready for analysis\n",
    "\n",
    "To perform the sentiment analysis for financial news we're going to load a local copy of this dataset: https://www.kaggle.com/datasets/ankurzing/sentiment-analysis-for-financial-news.\n",
    "\n",
    "This dataset contains two columns, `Sentiment` and `Sentence`. The sentiment can be `negative`, `neutral` or `positive`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "df = pd.read_csv(\"./datasets/sentiments_with_predictions.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run the model documentation tests\n",
    "\n",
    "First, use the ValidMind Library to initialize the dataset and model objects necessary for documentation. The ValidMind `predict_fn` function allows the model to be tested and evaluated in a standardized manner:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vm_test_ds = vm.init_dataset(\n",
    "    dataset=df,\n",
    "    input_id=\"test_dataset\",\n",
    "    text_column=\"Sentence\",\n",
    "    target_column=\"Sentiment\",\n",
    ")\n",
    "\n",
    "vm_model = vm.init_model(\n",
    "    model=FoundationModel(\n",
    "        predict_fn=call_model,\n",
    "        prompt=Prompt(\n",
    "            template=prompt_template,\n",
    "            variables=prompt_variables,\n",
    "        ),\n",
    "    ),\n",
    "    input_id=\"gpt_35_model\",\n",
    ")\n",
    "\n",
    "# Assign model predictions to the test dataset\n",
    "vm_test_ds.assign_predictions(vm_model, prediction_column=\"gpt_35_prediction\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, use the ValidMind Library to run validation tests on the model. The `vm.run_documentation_tests` function analyzes the current model's documentation template and collects all the tests associated with it into a test suite.\n",
    "\n",
    "The function then runs the test suite, logs the results to the ValidMind API and displays them to you.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_suite = vm.run_documentation_tests(\n",
    "    inputs={\n",
    "        \"dataset\": vm_test_ds,\n",
    "        \"model\": vm_model,\n",
    "    }\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Next steps\n",
    "\n",
    "You can look at the results of this test suite right in the notebook where you ran the code, as you would expect. But there is a better way: view the prompt validation test results as part of your model documentation in the ValidMind Platform:\n",
    "\n",
    "1. In the ValidMind Platform, go to the **Documentation** page for the model you registered earlier. ([Need more help?](https://docs.validmind.ai/guide/model-documentation/working-with-model-documentation.html))\n",
    "\n",
    "2. Expand **2. Data Preparation** or **3. Model Development** to review all test results.\n",
    "\n",
    "What you can see now is a more easily consumable version of the prompt validation testing you just performed, along with other parts of your model documentation that still need to be completed.\n",
    "\n",
    "If you want to learn more about where you are in the model documentation process, take a look our documentation on the [ValidMind Library](https://docs.validmind.ai/developer/validmind-library.html).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Upgrade ValidMind\n",
    "\n",
    "<div class=\"alert alert-block alert-info\" style=\"background-color: #B5B5B510; color: black; border: 1px solid #083E44; border-left-width: 5px; box-shadow: 2px 2px 4px rgba(0, 0, 0, 0.2);border-radius: 5px;\">After installing ValidMind, you’ll want to periodically make sure you are on the latest version to access any new features and other enhancements.</div>\n",
    "\n",
    "Retrieve the information for the currently installed version of ValidMind:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip show validmind"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If the version returned is lower than the version indicated in our [production open-source code](https://github.com/validmind/validmind-library/blob/prod/validmind/__version__.py), restart your notebook and run:\n",
    "\n",
    "```bash\n",
    "%pip install --upgrade validmind\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You may need to restart your kernel after running the upgrade package for changes to be applied."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
