December 26, 2024
THIS WAS AUTOMATICALLY GENERATED BY THE NEW RELEASE NOTEBOOK.
Release highlights
Move capital markets notebooks from codesharing to codesamples
This notebook demonstrates the support capital markets models in ValidMind.
Generated PR summary:
This update brings several improvements to Jupyter notebooks and Python modules related to option pricing models and JSON encoding. In the notebooks, the ‘Data Preparation’ section is removed from quickstart_option_pricing_models.ipynb, a typo in a benchmark test description is corrected, code cell execution counts are adjusted for consistency, and the method for accessing data for plotting results is updated. Additionally, unnecessary comments are removed, code formatting is enhanced, and two redundant notebooks on option pricing models and sensitivity analysis are deleted. In terms of Python modules, metadata key formatting in validmind/tests/comparison.py is simplified by removing certain prefixes. Meanwhile, the NumpyEncoder class in validmind/utils.py is refactored for better readability and maintainability through the introduction of type handler methods for various data types. These updates aim to enhance clarity, maintainability, and functionality within the codebase.
Enhancements
Update RAG documentation demo notebook
Updated RAG documentation demo notebook
Added section on generation quality and included the following tests:
Token_DisparityRouge_ScoreBleu_ScoreBert_ScoreMeteor_Score
Added section on bias and toxicity and the following tests:
Toxicity_ScoreRegard_Score
Generated PR summary:
This update to the RAG documentation demo notebook introduces new tests and enhances logging functionality. You now have detailed instructions for initializing the client library with the ValidMind code snippet, aiding in model registration and connection. The punkt_tab resource is downloaded to support tokenization tasks.
Logging is improved by appending the .log() method to several test invocations for better result tracking. New tests evaluate the quality of RAG model outputs, including cosine similarity distribution, token disparity, and various scoring measures like ROUGE, BLEU, BERT, and METEOR. Toxicity and regard scores are added to identify harmful language and sentiment bias.
Additionally, the kernel specification for the notebook is updated to use Python 3.10. These enhancements aim to improve the robustness of RAG model evaluations, aligning outputs better with references and identifying potential biases or quality issues.
Add parameter grid support in the comparison tests functionality
We support for comparison tests using the input_grid parameter in run_test functionality. However, similar support is required for params. This is useful for any case where you want to run the same test against multiple combinations of test parameters. You can create a single documentation block that compares the individual results.
The updated run_test() function now allows you to pass an param_grid. It runs a test for all combinations of parameters.
Example - for this param grid:
param_grid = {
"param1": [1],
"param2": [0.1, 0.2],
}A test runs once for each of the following param groups:
{"param1": 1, "param2": 0.1}
{"param1": 1, "param2": 0.2}Generated PR summary:
This update introduces enhancements and bug fixes to the ValidMind test framework and its Jupyter notebooks. In the quickstart_regression_full_suite.ipynb notebook, you now use the project parameter instead of model in the vm.init function, aligning with the new API structure. In 2_run_comparison_tests.ipynb, a new test is available for running comparisons with multiple parameter values using param_grid.
The code now includes an improved ClassifierPerformance class that features a default_params attribute and supports the average parameter for ROC AUC calculations. The run.py script has been refactored to allow parameter grids (param_grid) alongside input grids (input_grid) for running comparison tests, providing more flexible test configurations.
Additional helper functions validate test inputs and grid configurations, enhancing test execution robustness. Unnecessary blank lines have also been removed from several Python files to improve readability. These updates aim to increase the flexibility and functionality of the ValidMind testing framework, enabling more comprehensive and varied testing scenarios.
Added the ability to manage saved views
New feature allows you to save filters, sorting, and columns as predefined Views for Model Inventory and Findings.
Generated PR summary:
This update introduces a feature for managing saved views within the application. You can now use enhanced API functions to handle saved views, including fetching, creating, updating, and deleting them. A new component, ManageViewsButton, allows you to manage these views through a user interface that provides options to add or edit views using a modal dialog, delete views with confirmation, and display and select from a list of saved views. The ManageViewsButton component is integrated into the ModelFindings and ModelInventory pages, enabling you to save and manage views related to findings and models. To control this functionality, a new feature flag, modelInventorySavedViews, has been introduced. Additionally, a model called TSavedView has been defined to represent saved views with properties such as cuid, name, description, type, and content.
Mnt analytics improvements
Changes to analytics:
Added sorting capability for bar charts.
Added a new action for metrics, Count %. This lets you count the percentage of a given metric.
Generated PR summary:
This update enhances the AddOrEditVisualizationModal component in the web application. It introduces a new sorting functionality with the addition of a SortingComponent, which allows you to sort visualizations based on selected metrics or groupings. This component dynamically generates sorting options using the useMemo hook, considering the current dataset, metrics, and grouping. Additionally, a ‘Count %’ option is now available in the MetricSelector for string, number, date, and boolean metric types, enabling you to view metrics as percentages. Minor UI adjustments have been made for a better user experience; these include changing the overflow property of ModalBody to hidden and adding vertical scrolling (overflowY: 'auto') to the VStack. These improvements aim to provide more flexibility in data visualization and enhance interaction with your data.
chore
add UI changes for revision history
- Implemented UI enhancements for the revision history feature.
Generated PR summary:
This update introduces the RevisionHistoryButtonPlugin to enhance the CKEditor component within the CKEditorWrapper. This new feature allows you to navigate revision history more effectively by adding a button to the editor’s interface. Upon initialization, the plugin integrates these buttons into specific UI elements, such as the toolbar and sticky panel.
Additionally, there are comprehensive CSS enhancements designed to improve dark mode styling for CKEditor and its components. These modifications ensure visual consistency and accessibility in dark mode by adjusting background colors, text colors, borders, and other styling properties to align with the dark theme.
Adding custom field permission check
Administrators can now assign write permissions to individual fields in the model inventory.
Generated PR summary:
This update introduces several enhancements to custom field permissions management within the application. Permission handling is simplified by replacing the TPermissionAction type with a generic string[] type across various components. New states and logic are added in the Custom Field Modal for managing role changes, allowing you to add or remove roles with specific permissions more easily. A new useCustomFieldPermissions hook helps fetch and manage these roles efficiently.
The onSaveClicked function now handles role permission updates asynchronously, providing feedback through toast notifications. User interface improvements include integrating chakra-react-select for a smoother multi-select experience in role management and enhancing the modal state on the Custom Fields page to ensure seamless transition between editing and adding new fields.
Context and hooks have been updated, including changes to InventoryModelContext and UsersContext, to use the new permission type. A utility function, isReadOnlyField, has been introduced in the Inventory Model Overview to determine if a field is editable based on permissions. These updates enhance flexibility and usability when managing custom field roles and permissions.
Bug fixes
Deleting a finding
Fixed an issue where users could delete findings even if they were assigned to a validation report.
Generated PR summary:
This update enhances the web application by adding a new function, GetFindingLinkedAssessmentFindings, to the API module. This function retrieves related assessment findings, allowing you to view or delete a finding while being aware of its linked assessments. The AssessmentFinding and Assessment models now include optional fields for assessment and inventory_model, offering more detailed information about each finding and assessment. In the user interface, the ViewFinding component displays a warning alert when you attempt to delete a finding associated with any assessments. This alert lists the affected assessments to help you understand the consequences of your action. Additionally, the ConfirmationAlert component has been enhanced to accept more complex dialog content through its updated dialogBody prop. Overall, these improvements provide better context and warnings when managing findings, ensuring you are informed of potential impacts on linked assessments.
Documentation
Actions to dynamically generate an .env file
Valid Mind Academy
The Valid Mind Academy Developer Fundamentals course has been updated. It includes an improved version of the Valid Mind Introduction for Model Developers Jupyter Notebook.
Now this embedded notebook is executed live within the training. You can interact with output cells that were previously omitted, such as when previewing the documentation template.
This training notebook serves as a reference guide. It helps users understand what to expect when they first run the cells.
Jupyter notebooks
We’ve now included details on how to initialize Valid Mind with credentials stored in an
.envfile within our Jupyter Notebook samples.Our documentation guides have also been updated to match this new experience.
Generated PR summary:
This update enhances the environment configuration and execution of Jupyter Notebooks in your project. You now have new environment variables in the .env.example file, including VM_API_HOST, VM_API_KEY, VM_API_SECRET, and VM_API_MODEL. GitHub Actions for various notebook environments (demo-notebook, prod-notebook, staging-notebook) now require an env_file input to ensure that the .env file is present before notebooks are executed. Workflow files have been updated to dynamically create a .env file using secrets.
A new execute target has been added to the Makefile, allowing you to execute Jupyter Notebooks with specified profiles and paths. The process involves duplicating notebooks for execution rather than checking them out from the main branch.
Documentation has been improved to provide detailed instructions on storing model credentials in .env files and their usage within notebooks. Additionally, it includes examples and explanations for enabling monitoring when using these files. These enhancements aim to improve flexibility, security, and streamline the execution and monitoring processes for Jupyter Notebooks.
Add developer fundamentals videos & training fixes
We created a series of ten short videos
We created a series of ten short videos to help you learn about the model documentation process as a developer.
These videos introduce you to generating model documentation that includes tests. You also learn about adding your own tests, editing the content online, and then submitting your documentation for approval so that a validator can review it.
Watch the video series Title: Developer fundamentals
Generated PR summary:
This update to the ValidMind project includes several enhancements and adjustments. The titles of training modules in the internal/templates/videos/index.qmd file have been revised, renaming “Documenting Models 101” to “Developer Fundamentals” and “Validating Models 101” to “Train a model.” You’ll find three new SVG assets added in the site/assets/img/ directory with unique colors: deep green, light green, and pink. For guides, there’s a new video series titled ‘Developer Fundamentals’ with 10 videos introduced in the site/guide/guides.qmd file, along with a card title added for ‘Validating Models 101.’ In notebooks, all references to the “ValidMind Developer Framework” are now updated to “ValidMind Library,” impacting sections related to initialization and documentation across different processes. The training modules in site/training/developer-fundamentals/developer-fundamentals.qmd see updates that enhance interactivity and correct section references. Lastly, a new JSON file named test-results/.last-run.json has been introduced to track test results, confirming that all tests have passed.
Other
Chore
Undo some testing changes
Generated PR summary:
This update to the CI workflow configuration introduces two significant changes aimed at improving efficiency and responsiveness. First, the workflow now utilizes ubuntu-latest instead of a specific version, ensuring it consistently operates on the most current stable release of Ubuntu. This adjustment enhances compatibility with the latest software and security updates. Second, failure notifications via Slack are now enabled, allowing for immediate alerts through a Slack webhook if any issues arise during the integration process. This change helps you quickly address and resolve potential problems in the CI workflow.
Fix main history
Generated PR summary:
This release brings several enhancements and bug fixes to the ValidMind Library. The GitHub workflow is updated to ignore changes in the docs/_build/** path for the main and release-v1 branches, reducing unnecessary workflow runs. In the notebooks, the Python environment initialization is improved in 2_run_comparison_tests.ipynb by adding necessary imports and setting up the environment for data analysis. The import of xgboost and %matplotlib inline is moved earlier to ensure availability when needed. A minor bug fix corrects a print statement in run_unit_metrics.ipynb.
Code refactoring includes removing validmind/tests/metadata.py with its functionality integrated into validmind/tests/load.py. The file validmind/tests/comparison.py now includes a new _handle_metrics function for better metric handling from multiple test results. Changes in validmind/tests/run.py improve metadata collection and test run handling, while utility functions are reorganized into a new utils.py.
The Python version in Jupyter notebook metadata updates from 3.8.13 to 3.10.13, along with version changes in both pyproject.toml and __version__.py, bumping from 2.6.0 to 2.6.1.
Dependency updates include modifications in the poetry.lock file with new package versions and additional files for various platforms, enhancing functionality, maintainability, and performance of the ValidMind Library.
Chore: undo some testing changes
Generated PR summary:
The CI workflow configuration has been updated with two key improvements. Firstly, it now uses ubuntu-latest instead of a specific version (ubuntu-24.04). This ensures that your workflow always runs on the latest stable version of Ubuntu, helping to maintain compatibility with the most recent software and security updates. Secondly, failure notifications have been enabled by uncommenting the section that uses a Slack webhook. This change allows you to receive alerts in case of a failure during the integration job, enhancing your team’s ability to respond promptly to any issues in the CI process.
B
u m p
c r o s s-spawn
How to upgrade
ValidMind Platform
To access the latest version of the ValidMind Platform,1 hard refresh your browser tab:
- Windows:
Ctrl+Shift+RORCtrl+F5 - MacOS:
⌘ Cmd+Shift+ROR hold down⌘ Cmdand click theReloadbutton
ValidMind Library
To upgrade the ValidMind Library:2
In your Jupyter Notebook:
Then within a code cell or your terminal, run:
%pip install --upgrade validmind
You may need to restart your kernel after running the upgrade package for changes to be applied.