January 1, 2025

THIS WAS AUTOMATICALLY GENERATED BY THE NEW RELEASE NOTEBOOK.

Relative link to training

Release highlights

Move capital markets notebooks from codesharing to codesamples

This notebook demonstrates the support capital markets models in ValidMind.

Generated PR summary:

This update introduces several changes across Jupyter notebooks and Python modules related to option pricing models and JSON encoding.

For the notebooks, the ‘Data Preparation’ section is removed from quickstart_option_pricing_models.ipynb, a typo in the benchmark test description is corrected, and code cell execution counts are adjusted for consistency. The method of accessing data for plotting results is updated, redundant comments are removed, and code formatting is improved. Additionally, two notebooks—OptionPricer.ipynb and option_pricing_models_vm.ipynb—are deleted as they contained implementations of option pricing models and sensitivity analysis that are no longer necessary.

In the Python modules, metadata key formatting in validmind/tests/comparison.py is simplified by removing unnecessary prefixes. The NumpyEncoder class in validmind/utils.py is refactored to enhance readability and maintainability with new type handler methods for different data types, such as datetime, pandas intervals, numpy data types, and QuantLib dates.

These updates enhance clarity, maintainability, and functionality within the context of option pricing models and JSON data handling.

Enhancements

Added the ability to managed saved views

New feature allows you to save filters, sorting, and columns as predefined views for model inventory and findings.

Generated PR summary:

This update introduces a new feature for managing saved views within the application. You now have access to enhanced API functions that allow you to handle saved views, such as fetching, creating, updating, and deleting them based on type. A new UI component, ManageViewsButton, has been added to facilitate the management of your saved views. This component enables you to add or edit views using a modal dialog, delete views with a confirmation prompt, and display and select from a list of saved views.

The integration of the ManageViewsButton extends to both the ModelFindings and ModelInventory pages. This allows you to save and manage views specifically related to findings and models in these sections. Additionally, there is a new feature flag named modelInventorySavedViews, which is available for toggling the functionality related to saved views. A model definition called TSavedView has also been introduced; it includes properties such as identification (cuid), name, description, type, and content for representing saved views comprehensively.

Mnt analytics improvements

Changes to analytics

  • Added sorting capability for bar charts.

  • Added a new action for metrics, count %. Lets you count the percentage of a given metric.

Generated PR summary:

This update enhances the AddOrEditVisualizationModal component in the web application by adding several new features. You can now sort visualizations through a new SortingComponent, which allows sorting based on selected metrics or groupings. This feature uses the useMemo hook to offer dynamic sorting options tailored to your current dataset, metrics, and grouping. Additionally, the MetricSelector includes an option for ‘Count %’ across various metric types such as string, number, date, and boolean, enabling you to view metrics as percentages. Minor UI adjustments have been made for a better user experience; these include modifying the ModalBody overflow property to hidden and adding a vertical scrolling feature with overflowY: 'auto' in the VStack. These improvements aim to provide you with greater flexibility in viewing and interacting with your data.

Add UI changes for revision history

Implemented ui enhancements for the revision history feature.

Generated PR summary:

This update introduces the new RevisionHistoryButtonPlugin to the CKEditor component in the CKEditorWrapper. The plugin adds a button to the editor’s interface, enabling you to navigate revision history more efficiently. It integrates with existing UI elements like the toolbar and sticky panel once the editor is ready.

Additionally, this release includes significant CSS enhancements for dark mode styling. These improvements ensure that CKEditor and its components, including toolbars, buttons, and revision history sidebars, are visually consistent and accessible when using a dark theme. Adjustments have been made to background colors, text colors, borders, and other styling properties to fit well within the dark mode aesthetics.

Adding custom field permission check

Administrators can now assign write permissions to individual fields in the model inventory.

Generated PR summary:

This update enhances the management of custom field permissions within the application. The TPermissionAction type is refactored to a generic string[] type, streamlining permission handling across various components. You can now manage role changes for custom fields more effectively with new states and logic, allowing you to add or remove roles with specific permissions. A new useCustomFieldPermissions hook fetches and manages these roles. The updated onSaveClicked function handles role permission updates asynchronously, providing feedback through toast notifications.

UI improvements include integrating chakra-react-select for an enhanced multi-select experience when managing roles in the CustomFieldModal. The CustomFields page now handles modal states more efficiently, using a single instance for both editing and adding new fields.

The update also modifies contexts and hooks by updating the InventoryModelContext and UsersContext to use the new permission type. A utility function called isReadOnlyField is added in the `InventoryModelOverview’ to assess field editability based on permissions. These enhancements aim to improve flexibility and usability in managing custom fields with role-based permissions.

Update RAG documentation demo notebook

Updated RAG documentation demo notebook

  • Added section on generation quality and included the following tests:

    • Token disparity
    • Rouge score
    • Bleu score
    • Bert score
    • Meteor score

  • Added section on bias and toxicity and the following tests:

    • Toxicity score
    • Regard score

Generated PR summary:

This update enhances the RAG documentation demo notebook with new tests and improved logging functionality. You will find expanded instructions on initializing the client library, which now include a detailed guide on obtaining and using the ValidMind code snippet for model registration and connection. The punkt_tab resource is also downloaded to support tokenization tasks.

Logging is improved by appending the .log() method to several test invocations, ensuring results are properly logged. New tests are introduced to assess the quality of RAG model outputs, including evaluations of cosine similarity distribution, token disparity, and various text generation quality scores like ROUGE, BLEU, BERT, and METEOR. Additionally, toxicity and regard scores are assessed to identify harmful language or sentiment bias.

The notebook’s kernel specification is updated to use Python 3.10. These changes aim to enhance the robustness of the RAG model evaluation process by improving alignment with reference outputs and identifying potential biases or quality issues.

Add parameter grid support in the comparison tests functionality

We support comparison tests using the input_grid parameter in run_test functionality. However, similar support is needed for params. You may find this useful when you want to run the same test against multiple combinations of test parameters. It helps create a single documentation block that compares individual results.

The updated run_test() function now allows you to pass a param_grid. The test runs for all combinations of parameters.

Example param grid:

param_grid = {
    "param1": [1],
    "param2": [0.1, 0.2],
}

A test runs once for each of the following param groups:

{"param1": 1, "param2": 0.1}
{"param1": 1, "param2": 0.2}

Generated PR summary:

This update to the ValidMind test framework includes enhancements and bug fixes, along with updates to the associated Jupyter notebooks. In the quickstart_regression_full_suite.ipynb notebook, the vm.init function now uses a project parameter instead of model, and parameters align with the new API structure. The 2_run_comparison_tests.ipynb notebook introduces a new test for running comparisons with multiple parameter values using param_grid.

The ClassifierPerformance class now features a default_params attribute and supports an average parameter for ROC AUC calculations. Updates to the run.py script enable it to handle parameter grids (param_grid) alongside input grids (input_grid) for more flexible comparison tests. Additionally, helper functions have been incorporated to validate test inputs and grid configurations, enhancing the robustness of test execution.

Finally, unnecessary blank lines have been removed from several Python files, improving code readability. These changes enhance the flexibility and functionality of the ValidMind testing framework for more comprehensive testing scenarios.

Bug fixes

Deleting a finding

should remove the finding from a validation report

Fixed an issue where users could delete findings even if they were assigned to a validation report.

Generated PR summary:

This update enhances the web application by adding a new function, GetFindingLinkedAssessmentFindings, to the API module. This function allows you to retrieve linked assessment findings for a given finding, enabling the display of related assessments when viewing or deleting findings. The AssessmentFinding and Assessment types in the guideline.ts model now include optional fields for assessment and inventory_model, offering more detailed information associated with each finding and assessment.

The user interface has been improved with the inclusion of a warning alert in the ViewFinding component. When you attempt to delete a finding linked to one or more assessments, the alert lists the assessments that will be affected, helping you understand the impact of your actions. Additionally, the ConfirmationAlert component’s dialogBody prop now accepts ReactNode, allowing for more complex and informative dialog content. These updates collectively enhance your experience by providing context and warnings regarding potential impacts on linked assessments when deleting findings.

Documentation

Actions to dynamically generate an .env file

Valid Mind Academy

  • The Valid Mind Academy Developer Fundamentals course now features an updated version of the Valid Mind Introduction for Model Developers Jupyter Notebook.

  • The embedded notebook executes live within the training. You can interact with output cells that were previously omitted, such as when previewing the documentation template.

  • This training notebook also serves as a reference guide. It shows you what to expect when you first run the cells.

Jupyter notebooks

  • We’ve added details on how to initialize Valid Mind using credentials stored in an .env file within our Jupyter Notebook samples.

  • Our documentation guides have been updated to match this new experience.

Generated PR summary:

This update brings enhancements to the environment configuration and execution of Jupyter Notebooks within your project.

Firstly, there are several updates to the environment configuration. The .env.example file now includes new environment variables: VM_API_HOST, VM_API_KEY, VM_API_SECRET, and VM_API_MODEL. Adjustments have been made to GitHub Actions (demo-notebook, prod-notebook, staging-notebook) requiring an env_file input, ensuring that the .env file is available before running notebooks. Additionally, workflow files (deploy-docs-prod.yaml, deploy-docs-staging.yaml, validate-docs-site.yaml) dynamically create a .env file using secrets prior to notebook execution.

For notebook execution, a new execute target has been added in the Makefile. This helps you run Jupyter Notebooks with specified profiles and file paths more easily. Furthermore, the process for executing notebooks has been optimized by duplicating them in the Makefile instead of checking them out from the main branch.

Lastly, documentation has been updated to provide clearer guidance on storing model credentials in .env files and their utilization within notebooks. There are now examples and explanations in place for enabling monitoring when using .env files, offering better support for your documentation needs.

These improvements aim to enhance the flexibility and security of managing environments while streamlining how you execute and monitor Jupyter Notebooks.

Add developer fundamentals videos & training fixes

we created a series of 10 short videos to help you learn about the model documentation process as a developer. these videos introduce you to generating model documentation that includes tests, adding your own tests, editing the content online, and then submitting your documentation for approval so that a validator can review your work.

Generated PR summary:

This update introduces several enhancements and updates to the ValidMind project. The titles of training modules in the internal/templates/videos/index.qmd file have been modified, with “Documenting Models 101” renamed to “Developer Fundamentals” and “Validating Models 101” changed to “Train a model”. New SVG files, including inputoutput-deepgreen.svg, inputoutput-lightgreen.svg, and inputoutput-pink.svg, have been added to the assets directory for visual elements. A new video series titled ‘Developer Fundamentals’, comprising 10 videos, is now available in the guide section, while a card title has been added to the ‘Validating Models 101’ series. References within the tutorial notebook for model developers have been updated from “ValidMind Developer Framework” to “ValidMind Library”, affecting several sections like initialization and documentation processes. In the training module content, section references have been corrected, and interactivity has been improved. Finally, a new JSON file has been introduced to track test results, confirming all tests have passed successfully.

Add login button to docs site

Option to log in

You can now log in to the ValidMind platform from our docs site. This makes it easy for you to find the right login. Click Login to try logging in yourself.

Generated PR summary:

This update enhances the website’s navigation menu by adding a ‘Login’ section with multiple login options. The search box is removed from the sidebar to streamline user experience. A new ‘Login’ menu appears on the right side of the navigation bar, providing links to different login portals (US1 and CA1) for the ValidMind platform. Additionally, a help link titled ‘Which login should I use?’ directs you to a guide for logging into ValidMind. The styling in styles.css is updated to support this new login menu, including adjustments to background color, border, text color, padding, and hover effects.

Rework training section and update validator training

We introduced a new training homepage. It is easier to find and register for courses now. We also expanded our validator training and user guide with more guidance.

You can now track and resolve issues identified in model findings more effectively.

training Training

Generated PR summary:

This release includes enhancements and bug fixes to the ValidMind documentation and training templates. You now have access to new course templates for ‘Administrator Fundamentals’, ‘Developer Fundamentals’, and ‘Validator Fundamentals’. These templates offer structured learning paths, complete with learning objectives, course content, registration details, and support via Slack and email.

The documentation has been updated for improved clarity and consistency. Instructions such as “Select a model by clicking on it” have been refined to guide you better in finding your model by applying a filter or searching. Navigation links and file paths are updated to match the new course structures.

Site navigation has been modified to incorporate new training paths, with adjustments made in _quarto.yml and iframe links in index.qmd updated to direct you to the new course pages.

A new section for tracking issue resolution during the validation process is introduced. This provides guidance on assessing open issues and determining the severity of findings.

These updates aim to enhance your experience through clearer guidance and structured learning tailored to different user roles.

Document LLM features

ValidMind offers several specialized features that use large language models (LLMs) to streamline model risk management and ensure regulatory compliance. Here’s how we approach these features and what you need to know:

Large language model features

Generated PR summary:

This update brings several enhancements and new features to the project. The preview functionality has been improved by updating the CSS for the .preview class, allowing dynamic sizing without fixed width and height. You can now use optional width and height attributes in the Lua script, defaulting to 400 and 225, with external URL handling for source and target. The HTML content generated for the preview div ensures that the iframe scales to fit its dimensions.

A new documentation file, overview-llm-features.qmd, details the large language model (LLM) features available in the product. This document discusses aspects such as test interpretation, risk assessment, qualitative checks, and document checker functionalities while highlighting a focus on using company products for risk management and compliance. The site configuration now includes a navigation link to this new documentation.

Additionally, new media files have been added to support LLM feature documentation, including images and a GIF. These updates aim to enhance preview feature flexibility and offer detailed insights into the LLM capabilities of your product.

Other

Bump cross-spawn

Undo some testing changes

Generated PR summary:

This update to the CI workflow configuration includes two significant changes. First, the workflow now uses ubuntu-latest instead of a specific version (ubuntu-24.04). This ensures that it always runs on the latest stable version of Ubuntu, helping maintain compatibility with recent software and security updates. Second, failure notifications are now enabled with a Slack webhook. This enhancement improves your team’s ability to quickly respond to and address issues during the integration process by sending notifications if any failures occur.

Fix main history

Ensure you provide a clear and concise description. Describe the function or purpose of the content you are working on.

Generated PR summary:

This update introduces several enhancements and bug fixes to the ValidMind Library. The GitHub workflow is improved by updating the docs.yaml workflow file to ignore changes in the docs/_build/** path during pushes to the main and release-v1 branches, reducing unnecessary workflow runs. For notebook updates, the initialization of the Python environment in 2_run_comparison_tests.ipynb is enhanced with necessary imports for data analysis, and the import of xgboost and %matplotlib inline is now available earlier. In run_unit_metrics.ipynb, a print statement bug is corrected from result.scalar to result.metric. Code refactoring includes removing the validmind/tests/metadata.py file and integrating its functionality into validmind/tests/load.py, enhancing the _handle_metrics function in validmind/tests/comparison.py, improving metadata collection and test run handling in validmind/tests/run.py, and reorganizing utility functions by moving them to utils.py. The Python version in Jupyter notebook metadata is updated from 3.8.13 to 3.10.13, along with a version bump from 2.6.0 to 2.6.1 noted in both the pyproject.toml and __version__.py files. Additionally, there are updates to the poetry.lock file with new package versions for various platforms. These changes aim to enhance the functionality, maintainability, and performance of the ValidMind Library for you as a user.

Undo some testing changes

Generated PR summary:

This update to the CI workflow configuration introduces two important improvements. First, the runner version now uses ubuntu-latest, replacing a specific version (ubuntu-24.04). This change ensures that your workflow runs on the latest stable Ubuntu release, helping maintain compatibility with current software and security updates. Additionally, failure notifications have been enabled by activating a previously commented-out section. This feature uses a Slack webhook to notify you of any failures during integration jobs, allowing for quicker responses and issue resolution in the CI process.

How to upgrade

ValidMind Platform

To access the latest version of the ValidMind Platform,1 hard refresh your browser tab:

  • Windows: Ctrl + Shift + R OR Ctrl + F5
  • MacOS: ⌘ Cmd + Shift + R OR hold down ⌘ Cmd and click the Reload button

ValidMind Library

To upgrade the ValidMind Library:2

  1. In your Jupyter Notebook:

    • Using JupyterHub:3 Hard refresh your browser tab.
    • In your own developer environment:4 Restart your notebook.

  2. Then within a code cell or your terminal, run:

    %pip install --upgrade validmind

You may need to restart your kernel after running the upgrade package for changes to be applied.