Get started with the ValidMind Developer Framework

Published

September 4, 2024

The ValidMind Developer Framework helps you streamline model documentation by automating the generation of drafts. All you need to do is upload your documentation artifacts and test results to the ValidMind platform.

What is the ValidMind Developer Framework?

ValidMind’s developer framework provides a rich collection of documentation tools and test suites, from documenting descriptions of your dataset to validation testing your models for weak spots and overfit areas.

An image showing the two main components of ValidMind. The developer framework that integrates with your existing developer environment, and the ValidMind Platform UI.

ValidMind offers two primary methods for automating model documentation:

Generate documentation — Through automation, the developer framework extracts metadata from associated datasets and models for you and generates model documentation based on a template. You can also add more documentation and tests manually using the documentation editing capabilities in the ValidMind Platform UI.
Run validation tests — The developer framework provides a suite of validation tests for common financial services use cases. For cases where these tests do not cover everything you need, you can also extend existing test suites with your own proprietary tests or testing providers.

The developer framework is designed to be model agnostic. If your model is built in Python, ValidMind’s Python client library already provides all the standard functionality you might need without requiring your developers to rewrite any functions.

Key ValidMind concepts

model documentation: A structured and detailed record pertaining to a model, encompassing key components such as its underlying assumptions, methodologies, data sources, inputs, performance metrics, evaluations, limitations, and intended uses.

Within the realm of model risk management, this documentation serves to ensure transparency, adherence to regulatory requirements, and a clear understanding of potential risks associated with the model’s application.

validation report: A formal document produced after a model validation process, outlining the findings, assessments, and recommendations related to a specific model’s performance, appropriateness, and limitations. Provides a comprehensive review of the model’s conceptual framework, data sources and integrity, calibration methods, and performance outcomes.

Within model risk management, the validation report is crucial for ensuring transparency, demonstrating regulatory compliance, and offering actionable insights for model refinement or adjustments.

template, documentation template: Functions as a test suite and lays out the structure of model documentation, segmented into various sections and sub-sections. Documentation templates define the structure of your model documentation, specifying the tests that should be run, and how the results should be displayed.

ValidMind templates come with pre-defined sections, similar to test placeholders, including boilerplates and spaces designated for documentation and test results. When rendered, produces a document that model developers can use for model validation.

test: A function contained in the developer framework, designed to run a specific quantitative test on the dataset or model. Test results are sent to the ValidMind platform to generate the model documentation according to the template that is associated with the documentation.

Tests are the building blocks of ValidMind, used to evaluate and document models and datasets, and can be run individually or as part of a suite defined by your model documentation template.

metrics, custom metrics: Metrics are a subset of tests that do not have thresholds. Custom metrics are functions that you define to evaluate your model or dataset. These functions can be registered with ValidMind to be used in the platform.

In the context of ValidMind’s Jupyter Notebooks, metrics and tests can be thought of as interchangeable concepts.

inputs: Objects to be evaluated and documented in the developer framework. They can be any of the following:

model: A single model that has been initialized in ValidMind with vm.init_model(). See the Model Documentation or the for more information.
dataset: Single dataset that has been initialized in ValidMind with vm.init_dataset(). See the Dataset Documentation for more information.
models: A list of ValidMind models - usually this is used when you want to compare multiple models in your custom tests.
datasets: A list of ValidMind datasets - usually this is used when you want to compare multiple datasets in your custom tests. See this example for more information.

parameters: Additional arguments that can be passed when running a ValidMind test, used to pass additional information to a test, customize its behavior, or provide additional context.
outputs: Custom tests can return elements like tables or plots. Tables may be a list of dictionaries (each representing a row) or a pandas DataFrame. Plots may be matplotlib or plotly figures.
test suite: A collection of tests which are run together to generate model documentation end-to-end for specific use cases.

For example, the classifier_full_suite test suite runs tests from the tabular_dataset and classifier test suites to fully document the data and model sections for binary classification model use cases.

Getting started

After you sign up for ValidMind to get access, try one of our getting started guides:

Quickstart for model documentation

Welcome! Let’s get you started with the basic process of documenting models with ValidMind.

ValidMind Introduction for Model Developers

As a model developer, learn how the end-to-end documentation process works based on common scenarios you encounter in model development settings.

Learn how to run tests

ValidMind provides many built-in tests and test suites which make it easy for developers to automate their model documentation. Start by running a pre-made test, then modify it, and finally create your own test:

Run dataset based tests

Use the ValidMind Developer Framework’s run_test function to run built-in or custom tests that take any dataset or model as input. These tests generate outputs in the form of text, tables, and images that get populated in model documentation.

Implement custom tests

Custom tests extend the functionality of ValidMind, allowing you to document any model or use case with added flexibility.

Learn more: Run tests & test suites

Try the code samples

Our code samples showcase the capabilities of the ValidMind Developer Framework. Examples that you can build on and adapt for your own use cases include:

What’s next

After you have tried out the developer framework, continue working with your model documentation in the AI Risk Platform (platform UI) online. There, you can:

Work with documentation templates to customize them to your specific needs
Work with model documentation in the UI to make edits, collaborate with validators, and submit your model documentation for approval
Export your finalized model documentation

For more in-depth guides, check out our breakdown of your complete journey as a model developer with ValidMind:

Documenting models

Use the developer framework to generate model documentation, view the results and refine your documentation in the platform UI, and collaborate with your team to complete documentation and validation projects within the same interface.