January 1, 2025

THIS WAS AUTOMATICALLY GENERATED BY THE NEW MAKE COMMAND WHICH USES THE SAME SCRIPT AS THE NOTEBOOK.

Release highlights

Move capital markets notebooks from code sharing to code samples

This notebook demonstrates the support capital markets models in validmind.

Generated PR summary:

This update introduces changes across multiple Jupyter notebooks and Python modules focusing on option pricing models and JSON encoding. In the Jupyter notebooks, the ‘Data Preparation’ section is removed from quickstart_option_pricing_models.ipynb. A typo in a benchmark test description is corrected, and code cell execution counts are adjusted for consistency. The method to access data for plotting results is updated from result.metric.summary.results[0].data to result.tables[0].data. Redundant comments are removed, and code formatting is improved. Additionally, two notebooks, OptionPricer.ipynb and option_pricing_models_vm.ipynb, which contained implementations of option pricing models and sensitivity analysis, are deleted.

In the Python modules, enhancements include changes in validmind/tests/comparison.py, where metadata key formatting is simplified by removing “Input:” and “Param:” prefixes. In validmind/utils.py, the NumpyEncoder class is refactored for better readability and maintainability by introducing type handler methods for various data types like datetime objects, pandas intervals, numpy data types, and QuantLib dates.

These updates aim to enhance clarity, maintainability, and functionality within the codebase related to option pricing models and JSON data handling.

Enhancements

Update RAG documentation demo notebook

Updated RAG documentation demo notebook

  • Added section on generation quality and included the following tests:

    • Token_Disparity
    • Rouge_Score
    • Bleu_Score
    • Bert_Score
    • Meteor_Score

  • Added section on bias and toxicity and the following tests:

    • Toxicity_Score
    • Regard_Score

Generated PR summary:

This update enhances the RAG documentation demo notebook by introducing new tests and improving logging functionality. You now have expanded instructions for initializing the client library, including obtaining and using the ValidMind code snippet for model registration and connection. The punkt_tab resource is also downloaded to support tokenization tasks.

The logging improvements involve appending the .log() method to several test invocations to ensure proper recording of results. New tests are available to evaluate the quality of the RAG model’s outputs, including assessments of cosine similarity distribution, token disparity, ROUGE, BLEU, BERT, METEOR scores, toxicity levels, and regard scores. These assess various aspects of text generation quality such as semantic similarity and potential biases.

Additionally, the notebook’s kernel specification has been updated to use Python 3.10. This release aims to enhance the robustness and comprehensiveness of RAG model evaluations by ensuring better alignment with reference outputs and identifying areas for improvement in quality or bias detection.

Add parameter grid support in the comparison tests functionality

We support for comparison tests
Using the input_grid parameter in run_test functionality. However, similar support is required for params.

Support for params is useful when you want to run the same test against multiple combinations of test parameters and create a single documentation block that compares the individual results.

Run test function update

The updated run_test() function now allows you to pass a param_grid that runs a test for all combinations of parameters.

Example param grid:

param_grid = {
    "param1": [1],
    "param2": [0.1, 0.2],
}

For each of the following param groups, a test runs once:

{"param1": 1, "param2": 0.1}
{"param1": 1, "param2": 0.2}

Generated PR summary:

This update enhances the ValidMind test framework and its associated Jupyter notebooks with several improvements and fixes. In the quickstart_regression_full_suite.ipynb, the vm.init function is updated to use a project parameter, aligning with the new API structure. The notebook 2_run_comparison_tests.ipynb now includes a test for running comparisons with multiple parameter values using param_grid for specifying configurations.

The code enhancements include updating the ClassifierPerformance class to feature a default_params attribute and support the average parameter for ROC AUC calculations. Additionally, the run.py script is refactored to accommodate parameter grids (param_grid) alongside input grids (input_grid) for more flexible test configurations. Helper functions are introduced to validate test inputs and grid configurations, improving execution robustness.

Unnecessary blank lines in various Python files are removed to enhance code readability. These improvements aim to provide greater flexibility and functionality in conducting comprehensive test scenarios within the ValidMind framework.

Added the ability to manage saved views

New feature allows you to save filters, sorting, and columns as predefined Views for Model Inventory and Findings.

Generated PR summary:

This update introduces a new feature for managing saved views within the application. You can now use enhanced API functions to handle your saved views directly. These include fetching saved views by type, creating new ones, updating existing views, and deleting those you no longer need. We’ve added a ManageViewsButton to the user interface that allows you to manage your saved views. This component lets you add or edit views using a modal dialog, delete them with a confirmation prompt, and display and select from a list of your saved views.

The ManageViewsButton is integrated into both the ModelFindings and ModelInventory pages, enabling you to save and manage views related to findings and models effectively. A feature flag called modelInventorySavedViews is available to enable the saved views functionality. Additionally, there is now a TSavedView model defined for representing saved views with attributes such as cuid, name, description, type, and content.

## Mn_t_ analytics improvements

Changes to analytics

  • Added sorting capability for bar charts

  • Added a new action for metrics, count %. You can count the percentage of a given metric.

Generated PR summary:

This update enhances the AddOrEditVisualizationModal component, improving data visualization capabilities in your web application. You can now sort visualizations with a new SortingComponent, which allows sorting based on selected metrics or groupings. The component uses the useMemo hook to dynamically generate sorting options from the dataset, metrics, and grouping available to you.

Additionally, the MetricSelector includes an option for ‘Count %’, enabling you to view various metric types—such as string, number, date, and boolean—as percentages.

Lastly, minor user interface adjustments have been made to improve your experience. These include changing the ModalBody overflow property to hidden and adding overflowY: 'auto' to the VStack.

These enhancements provide you with greater flexibility in viewing and interacting with your data.

Chore

Add UI changes for revision history

Implemented UI enhancements for the revision history feature.

Generated PR summary:

This update introduces the RevisionHistoryButtonPlugin to the CKEditor component within the CKEditorWrapper. This plugin adds a button to the editor’s user interface, enabling you to navigate revision history more effectively. The plugin initializes when the editor is ready and locates specific UI elements such as the toolbar and sticky panel to integrate the revision history buttons seamlessly.

Additionally, extensive CSS modifications enhance dark mode styling for CKEditor and its components. These changes ensure that UI elements like toolbars, buttons, and revision history sidebars are visually consistent and accessible in dark mode. Adjustments include background colors, text colors, borders, and other styling properties to align with the dark theme.

Adding custom field permission check

Administrators can now assign write permissions to individual fields in the model inventory.

Generated PR summary:

This update enhances the management of custom field permissions within the application. It simplifies permission handling by replacing the TPermissionAction type with a generic string[] type across various components. You can now manage role changes for custom fields more effectively, adding or removing roles with specific permissions through new states and logic in the Custom Field Modal. The introduction of the useCustomFieldPermissions hook allows for streamlined fetching and management of roles with permissions. You receive feedback via toast notifications when updating role permissions asynchronously.

The user interface has also been improved to enhance your experience. The integration of chakra-react-select offers a better multi-select experience when managing roles in the Custom Field Modal. Additionally, modal state handling on the Custom Fields page is enhanced to ensure seamless editing and adding of new fields.

Context and hook updates are included as well, with updates to InventoryModelContext and UsersContext embracing the new permission type. A new utility function, isReadOnlyField, has been added to determine field editability based on permissions in the Inventory Model Overview.

These enhancements aim to provide you with greater flexibility and ease in managing custom field permissions, particularly regarding role-based responsibilities.

Bug fixes

Deleting a finding

You should remove a finding from a validation report when you delete it

Fixed an issue where users could delete findings even if they were assigned to a validation report.

Generated PR summary:

This update enhances the web application with several new features and improvements. You can now use a new API function, GetFindingLinkedAssessmentFindings, to retrieve linked assessment findings for a specific finding, enabling you to view or delete related assessments more effectively. The guideline.ts model has been updated; the AssessmentFinding and Assessment types now include optional fields for assessment and inventory_model, providing richer detail for each finding and assessment. The user interface sees an improvement in the ViewFinding component, which now displays a warning alert when you attempt to delete a finding linked to one or more assessments. This alert lists the affected assessments, giving you a clear understanding of how your deletion will impact them. Additionally, the ConfirmationAlert component has been refined; its dialogBody prop can now accept ReactNode, allowing for more complex dialog content. Overall, these changes enhance user experience by offering clearer context and warnings during operations involving linked assessments.

Documentation

Actions to dynamically generate an .env file

Valid Mind Academy

  • The ValidMind Academy Developer Fundamentals course has an updated version of the ValidMind Introduction for Model Developers Jupyter notebook.

  • Now, the embedded notebook executes live within the training. You can interact with output cells that were previously omitted, such as when previewing the documentation template.

  • This training notebook serves as a reference guide. It shows you what to expect when first running the cells.

Jupyter Notebooks

  • We now include details on initializing ValidMind with credentials stored in an .env file. These details are available in our Jupyter Notebook samples.

  • Our documentation guides have been updated to match this new experience.

Generated PR summary:

This update enhances environment configuration and the execution process of Jupyter Notebooks within the project. You can now find new environment variables, such as VM_API_HOST, VM_API_KEY, VM_API_SECRET, and VM_API_MODEL, added to the .env.example file. It ensures that GitHub Actions tied to notebooks (demo-notebook, prod-notebook, and staging-notebook) require an env_file input, confirming the presence of a .env file before notebook execution. Additionally, workflow files like deploy-docs-prod.yaml, deploy-docs-staging.yaml, and validate-docs-site.yaml are updated to dynamically create a .env file using secrets before running notebooks.

A new execute target is introduced in the Makefile, allowing you to run Jupyter Notebooks with specific profiles and paths. The Makefile now duplicates notebooks for execution instead of checking them out from the main branch.

Documentation has also been enhanced with detailed instructions on storing model credentials in .env files via updates to store-credentials-in-env-file.qmd. Moreover, examples and explanations for enabling monitoring when using .env files are added in enable-monitoring.qmd. These improvements enhance flexibility, security in environment management, and streamline running and monitoring Jupyter Notebooks.

Add developer fundamentals videos & training fixes

We created a series of 10 short videos to help you learn about the model documentation process as a developer. These videos introduce you to generating model documentation that includes tests, adding your own tests, editing the content online, and submitting your documentation for approval so that a validator can review it.

Generated PR summary:

This update introduces several enhancements to the ValidMind project aimed at improving your experience. The titles of training modules in the video templates have been revised, with “Documenting Models 101” renamed to “Developer Fundamentals” and “Validating Models 101” changed to “Train a model.” Three new SVG assets featuring specific colors are added for visual updates.

A new video series titled ‘Developer Fundamentals’ with ten videos is included in the guide enhancements, along with an updated card title for the ‘Validating Models 101’ series. References within the introductory tutorial for model developers have been updated from “ValidMind Developer Framework” to “ValidMind Library” across various sections. Additionally, training module adjustments enhance interactivity in documentation processes and correct section references.

Test results now include a JSON file tracking the status of the last test run, confirming that all tests have passed successfully.

Add login button to docs site

We added the option to log in to the ValidMind platform on our docs site. This makes it easy for you to find the right login. Click Login to try logging in.

Generated PR summary:

This update enhances the website’s navigation by adding a ‘Login’ section with multiple login options. The search box is removed from the sidebar to streamline the interface. You will now find a ‘Login’ menu on the right side of the navigation bar, which includes links to different login portals (‘US1’ and ‘CA1’) for accessing the ValidMind platform. Additionally, there is a help link titled ‘Which login should I use?’ that guides you on selecting the appropriate login portal for ValidMind. Styling updates in styles.css are included to support these changes, with adjustments to background color, border, text color, padding, and hover effects enhancing your overall user experience.

Other

chore: undo some testing changes

Generated PR summary:

This update to the CI workflow configuration involves two key changes. First, the CI workflow now runs on ubuntu-latest instead of a specific version like ubuntu-24.04. This adjustment ensures that you are always using the latest stable version of Ubuntu, maintaining compatibility with current software and security updates. Additionally, failure notifications are enabled using a Slack webhook to inform you of any failures during the integration job. This enhancement helps you quickly respond to and address issues in the CI process.

Fix main history

Generated PR summary:

This update to the ValidMind Library introduces several enhancements and bug fixes aimed at improving functionality, maintainability, and performance. The GitHub workflow for docs.yaml now ignores changes in the docs/_build/** path when pushing to the main and release-v1 branches, reducing unnecessary workflow runs. In the notebooks, improvements have been made for better initialization of the Python environment in 2_run_comparison_tests.ipynb, including moving the import of xgboost and %matplotlib inline to an earlier cell. A minor bug fix has been applied in run_unit_metrics.ipynb, correcting a print statement from result.scalar to result.metric.

Code refactoring includes integrating the functionality of the removed validmind/tests/metadata.py into validmind/tests/load.py. The file validmind/tests/comparison.py now features a new _handle_metrics function to handle metrics more effectively from multiple test results. Enhancements have also been made to improve metadata collection and test run handling in the validmind/tests/run.py. Additionally, utility functions have been moved from the module validmind/vm_models/result to a new file called utils.py.

Version updates include changing the Python version in Jupyter notebook metadata from 3.8.13 to 3.10.13 and updating both the pyproject.toml and __version__.py files to reflect a version bump from 2.6.0 to 2.6.1.

Finally, dependency updates are reflected with new package versions added in the updated poetry.lock, supporting various platforms.

Undo some testing changes

Generated PR summary:

This update to the CI workflow configuration includes two key changes. The workflow now uses ubuntu-latest instead of a specific version, ensuring that it always runs on the latest stable version of Ubuntu for better compatibility with current software and security updates. Additionally, failure notifications are enabled through a Slack webhook, allowing you to receive alerts for any failures during the integration job. This enhancement helps your team to quickly respond to and address issues in the CI process.

Bump cross-spawn

How to upgrade

ValidMind Platform

To access the latest version of the ValidMind Platform,1 hard refresh your browser tab:

  • Windows: Ctrl + Shift + R OR Ctrl + F5
  • MacOS: ⌘ Cmd + Shift + R OR hold down ⌘ Cmd and click the Reload button

ValidMind Library

To upgrade the ValidMind Library:2

  1. In your Jupyter Notebook:

    • Using JupyterHub:3 Hard refresh your browser tab.
    • In your own developer environment:4 Restart your notebook.

  2. Then within a code cell or your terminal, run:

    %pip install --upgrade validmind

You may need to restart your kernel after running the upgrade package for changes to be applied.